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Executive Summary 



In June of 1996, the Online Computer Library Center (OCLC), Office of Research, awarded 
a Library and Information Science Research Grant to Charles R. McClure and Steven K. Wyman, 
both of the School of Information Studies at Syracuse University, to explore the degree to which 
federal agencies' World Wide Web sites meet their constituencies’ needs. The research began in 
August, 1996, and concluded in April, 1997. 

Research Goals 

The goal of this study was to establish analytical tools based on both technical criteria and 
user feedback by which federal website administrators may assess the quality of their websites. The 
study combined qualitative and quantitative data collection techniques to achieve the following 
objectives: 

1. Identify and define key issues regarding representation and access to information 
resources through federal websites. 

2. Identify and analyze key information policy issues related to the design, development, and 
management of Web-based information resources and services. 

3. Establish evaluative criteria for reviewing federal websites 

4. Evaluate a sample of federal websites 

5. Suggest analytical tools with which federal website administrators can analyze site 
structure and use 

6. Offer recommendations to increase the usefulness of federal websites to libraries and 
other users. 

Progress toward these goals and objectives should provide a foundation for improving the design and 
operation of federal websites. 

Methodology 

This study was exploratory in nature and employed a combination of qualitative and 
quantitative techniques, providing both analytic and descriptive measures for reviewing federal 
websites. The methodology divided into four major categories: system-, user-, design-, and 

developer-based. Each category represented a distinct research approach to federal website 




viii 



7 



assessment, but the philosophy underlying the approach is that a system’s users ultimately determine 
the success of that system. In this vein, federal websites were treated as systems with respective user 
bases. Even though the user-based assessment formed the core of the study, the other three 
approaches all focus on the utility of a website to its constituency. 

Within the system-based method are website mapping and log file analysis. The study team 
used WebMapper (NetCarta), a commercial website management application, to gain structural 
surveys of the four federal websites being assessed. These surveys include error reports, applications 
operating from the site, internal and external links, gateways, and more. In order to gain a window 
into user movement through a website, log file analysis can be highly effective. 

The user-based approach consisted of situated assessments of four federal websites — the 
Department of Commerce, the National Archives and Records Administration, the National Institutes 
of Health, and Stat-USA. By “situated” we mean that each of the ten users (proxies) came to their 
assessment sessions with an interest in the content of the site which they reviewed. Either two or 
three proxies assessed the four websites. During one-hour assessments the researchers videotaped the 
computer monitor to preserve content context, and users’ comments as they narrated their evaluations 
according to the Think Aloud protocol. 

The design-based assessments divided into independent technical and policy evaluations 
conducted by three members of the study team. The design-based assessments, like the situated user 
assessments, followed scripted criteria. Technical elements included presentation / ease of use, 
content, usefulness, currency, navigation, help categories. 

The fourth and final assessment approach entailed a site visit to Stat-USA. The Stat-USA is 
unusual in that it maintains a subscription-based service tailored to high-end (volume) users. It 
operates on a cost-recovery basis and by its second year of service was breaking even. The 
experimental character of the Stat-USA website made it an attractive case study. Even though 
generalizations from such an example must be made carefully, the Stat-USA website administrators 
had clear views of the site’s critical success factors, as well as opinions about why a lack of 
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transferable to other websites is an open question. More study is required to examine costing issues 
of federal websites providing fee-based services. 

Significance of the Study 

It is useful to collect experiences of specific government website developers which may 
contribute to knowledge that can be more generally applied. Mistakes costly in terms of time and 
dollars might be avoided. Technical knowledge can enable other agencies to “leapfrog” certain 
hurdles. The data collection techniques and instruments developed and tested in this project can be 
used by federal Webmasters to improve their websites. The inclusion of user-based assessments are 
essential to developing high quality and useful websites. Thus, this research assists federal officials’ 
deployment and use of Web-based information resources and services. But by the same token, the 
critical success factors discovered through this study may be transferable to website designers in other 
settings. 

In this Web-based environment, however, there is the potential to provide a huge array of 
information products and services. The concept of Web-based information services is an evolving 
one. Assessment techniques, however, related to Web-based information sep/ices by specific 
audiences such as government officials, teachers, business persons, researchers, students, librarians, 
etc. are just in the development process. Thus, some key questions that will continue to be asked are: 

• What are the scope and reach of Web-based information services? 

• How much use do websites and individual pages receive and why are they used? 

• What types of users access the website and to what types of services and activities do they 
connect? 

• What are the costs for providing various types of Web-based services? 

• How has access to and use of Web-based services affected organizations and individuals in 
terms of economic competitiveness, education, productivity, quality of life, and other 
traditional performance indicators? 

• What types of performance measures are appropriate to determine the impact of Web-based 
services on various stakeholder groups? 

• What types of evaluation tools can assist policymakers website managers, and researchers 
measure those identified performance measures? 



To date, there has been little practical guidance offered to assess federal websites and Web-based 
services. As agencies implement new information technologies and expand Web-based services there 
will also be a need for evaluation methods and measures to demonstrate that the resources invested in 
such efforts have had some positive impact on the users they serve. 
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Chapter 1: Introduction 



Background 

The U.S. Government is the largest information provider in the world (Hemon, 1996: 1). The Gov- 
ernment Printing Office processes more than 1.6 million orders and ships over 110 million publications an- 
nually (Office of Technology Assessment, 1993: 2). Somewhere between 189,000 - 237,000 citizens use 
federal depository libraries each week, up from approximately 167,000 in 1989. From the GPO Access 
website alone, users download an average of 2.5 million documents per month from over 70 databases 
(Baldwin, November 1996: 3). In the average week around 30,000 academic and business researchers ob- 
tain research results from the National Technical Information Service. 

Electronic media permeate American society. A recently published report by Nielsen Media Research 
details the doubling in numbers of people in the U.S. and Canada using the Internet over an 18 month pe- 
riod ending January, 1997. 1 Increasingly, the federal government depends on digital systems to dissemi- 
nate information (McClure & Ryan, 1996: 297-313). Among the new media, the World Wide Web brings 
together diverse technical elements into a still fragile, but promising confederation (DePompa, 1997: 38). 
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dynamic platform on which publishing, communications, and data intermingle. 



Anyone with sufficient access and average computing resources can publish a website. No central author- 
ity controls what appears, no complete map to this new conceptual geography is yet possible, and no gen- 
erally recognized standards for consistent presentation direct the millions of pioneers through the new 
frontier. Even so, the allure of the Web entices individuals and institutions alike to stake claims in this 
terra incognita. U.S. federal website administrators are deploying a better means of presenting govern- 
ment information to a growing online audience (DiCaterino and Pardo, 1996). 

The growth in the number of federal websites is striking. The federal government began actively 
posting information on the Web in 1993. Now, nearly 900 federal websites may be accessed through a 
Web index at Villanova Center for Information Law and Policy, and this is not necessarily a complete list- 
ing <http://www.law.vill.edu/fed-agency/fedWebloc.html>. Yet, despite the rush by so many agencies to 



1 According to the Nielsen survey, approximately 50.6 million North Americans are accessing the Internet, 37.4 million of 
whom are using the Web. 



establish a Web presence, few guidelines exist for how to accomplish this in an “efficient and effective” 
manner as mandated in QMB Circular A- 130 regarding federal information resources and services (Office 
of Management and Budget, 1995). As this report was being prepared, the Office of Management and 
Budget issued another statement setting forth a set of principles for federal Web use (Katzen, 1997). There 
are a handful of preliminary guidance documents pertaining to individual agencies; the Office of Man- 
agement and Budget issued draft memoranda in July, 1996 (revised in November of 1996) suggesting 
guidelines for Executive Branch websites (Office of Management and Budget, July 16, 1996 and Novem- 
ber 17, 1996), and the NCSA/NSF World-Wide Web Federal Consortium in May of 1995 (updated No- 
vember 1996), posted guidelines online <http://skydive.ncsa.uiuc.edu/consortium/guide/hmpggl.htm>. A 
variety of additional policy documents relate to the use of electronic media for disseminating federal in- 



Policy analysts are just beginning to consider these and related policy instruments and how they will 
affect the production and dissemination of government information in a Web context. To date, there are 
few national guidelines or criteria to assist federal website administrators assess existing websites; nor is it 
clear to Webmasters how they may include user-based criteria for planning/designing site development 
(Tillman, 1996; Stoker & Cooke, 1995; Internet Business Network, 1995). Assessment criteria that do 
exist either are largely based on anecdotal information, or are directed at websites oriented to marketing 
initiatives (Snyder, 1996). Table 1 summarizes the major categories into which most website evaluation 
efforts fall. Examples of authors for each category are offered, as well. 



2 Portions of the Federal Records Act {44 U.S.C. Chapters 29, 3 1, 33, and 35), which pertain to agency dissemination of 
information through the Government Printing Office and the Federal Depository Library Program {44 U.S.C. 1902); 
relevant portions of. the Paperwork Reduction Act. {44 U.S.C. Chapter 35); and Electronic Freedom of Information Im- 
provement Act of 1996 (P.L. 104-231). Other regulations which potentially affect the development of federal websites 
include: Freedom of Information Act (5 U.S.C. 552), the Privacy Protection Act (5 U.S.C. 552a) or the Computer Security 
Act of 1987 {40 U.S.C. 759). Furthermore, the article does not include an analysis of issues associated with agencies’ 
charging for access to information, or the implications of the American Disabilities Act of 1990 {42 U.S.C. 12101) and the 
Rehabilitation Act Amendments of 1992 {29 U.S.C. 794); Establishment of Government Information Locator Service. 
(December 7, 1994). OMB Bulletin 95-01 . Washington, D C.: Office of Management and Budget; Application Profile 
for the Government Information Locator Service (GILS). (1994). FIPS PUB 192 Washington, D.C.: GPO. Note as well 
White House policy statements such as that of Vice President A1 Gore, Creating a Government That Works Better & Costs 
Less: Report of the National Performance Review. (September. 7, 1993). Washington, DC.: US. Government Printing 
Office; and the Information Infrastructure Task Force, The'National Information Infrastructure: Agenda for Action. 
(September 15, 1993). Washington, D.C.: National Telecommunications and Information Administration. 
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Federal websites are capable of providing fast, cost-effective access to an abundance of govern- 
ment information stored in a variety of electronic formats. The convergence of quick and relatively easy 
website publication with the federal government’s massive information output may at first glance seem 
harmonious. Often however, websites are built with little consideration of basic presentation standards and 
how such may affect the abilities of users to locate, trust and use public information provided online. Fur- 
thermore, it is important for agencies to provide information in a manner that best serves the interests of 
their users (Johnson, 1996). One of the best ways to achieve this result is by bringing the users into the 
process at an early phase in website development. 

Table 1. Overview of Selected Website Assessment Sources 



Library 

Science 

Perspective 


Since the evaluation of information resources and the 
organization of information is inherent in the field of 
Library and Information Science, much of the literature 
pertains to the assessment of information content and its 
organization in the website. 


Caywood, 1995; 
Pratt, Flanner and 
Perkins, 1996; 
Grassian, 1996; 
Tillman, 1996; 
Stoker and Cook, 
1995; Klobas, 1995; 
McClements and 
Becker, 1996 


Business 

Perspective 


The literature from the business field focuses on design- 
ing websites for customer use, and some of the literature 
discusses marketing and advertising issues. 


Internet Business Net- 
work, 1995; 
WebResults , 1996; 
Nash, 1995 


K-12 

Perspective 


Most of the educational literature on website assessment 
comes from the K-12 level and focuses on the quality of 
the information content provided on the website. 


McLachlan, 1996; 
Schrock, 1996 


Computer 

Science 

Perspective 


Except for the field of computer graphics, most of this 
literature from the field of computer science does not 
deal directly with website design. Many of the ideas 
presented within the computer science literature can be 
applied to the Web environment. 


Mitchell, 1996; 
Snyder, 1996; 

Udell, 1996; 
Schneiderman, 1987; 
Fidel, 1987 



Research Goals 

The. goal of this study was to establish analytical tools based on both technical criteria and user 
feedback by which federal website administrators may assess the quality of their websites. The research 
project began in August, 1996, and was completed in April, 1997. The study combined qualitative and 
quantitative data collection techniques to achieve the following objectives: 

1 . Identify and define key issues regarding representation and access to information resources through 
federal websites 
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2. Identify and analyze key information policy issues related to the design, development, and 
management of Web-based information resources and services 

3. Establish evaluative criteria for reviewing federal websites 

4. Evaluate a sample of federal websites 

5. Suggest analytical tools with which federal website administrators can analyze site structure and 
use 

6. Offer recommendations to increase the usefulness of federal websites to libraries and other users. 
Progress toward these goals and objectives should provide a foundation for improving the design and op- 
eration of federal websites. 

Assumptions and Limitations of the Study 

Discussions among the researchers during the selection of methods and design of structured in- 
struments showed that even within a small and more or less similar group, views could differ significantly 
over what was an appropriate approach for assessing websites. In part, the range of perspectives reflected 
differing concepts of what websites are, what they can or cannot do, and how best to devise a methodology 
which fairly represents the federal websites finally selected for assessment. Though this study does go 
some distance toward characterizing federal websites, it is still plain to the researchers that websites are 
chimerical. They may change suddenly and drastically. Multimedia features such as sound and video can 
radically alter the general experience of a website. Better conceptualizing websites, even though that may 
mean constructing an open-ended defmition, will continue to be important if researchers are to effectively 
study Web technology, organization, and use. 

Another area which requires attention is the nature of a website user base. It is presumed by the 
study team that users of a given website have some interest which overlaps the site content. This view led 
directly to the choice of a situated methodology augmented by the Think Aloud protocol. Despite the 
combined power of these two techniques, the research did not collect feedback from regular users. None- 
theless, it is our view that what this research intended to accomplish — namely to develop and test quality 
criteria for assessing federal websites - succeeded by using proxies. The proxies could be thought of as 
interested users of the websites they assessed. For most Web administrators, new users may serve as indi- 
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cators of new directions in information demand and use. New users may sometimes scrutinize a website 
quite carefully simply because they do not possess the experience of an environment that so often can blind 
one to the familiar. 

In the early stages of the research the study team selected appropriate websites to evaluate. The 
federal government has many websites, most operating without clear policy guidance. One good way to 
select websites is through log analysis. Log analysis can indicate webpages which are heavily trafficked, 
or have other features drawing or repelling users. However, to obtain agency log files as an external agent 
can be difficult. The researchers were very fortunate that Stat-USA provided a sample of their website’s 
use in the form of Accessjog files. As potent as log analysis can be, it can also drift into sensitive areas. 
Issues of user privacy must be handled carefully, and therefore the analysis of log files should be well 
thought out. For this study the logistics of arranging receipt of log file data collection and analysis meant 
that the analysis was not completed until the later phases of the study. Ideally, log analysis and website 
mapping should be done prior to technical and policy assessments. 

Regardless of the researchers’ awareness of the changeable character of websites, to conduct this 
study “snapshots” of the four selected websites provided the research data. This was not longitudinal, in 
other words. The quixotic nature of the Web environment motivated the researchers to use videotaping to 
preserve the context of the situated users’ assessments. Even so, by not including a longitudinal dimension 
in the research, significant changes in websites could go undetected. For instance, Stat-USA staff were in 
the process of comprehensively revising their website when this research began. They will have revised 
their site into a frame-based LotusNotes format by the time of this report, thus effectively addressing sev- 
eral shortcomings brought out by the assessment. 

The technical and policy assessments presented some challenges to the researchers, which will be 
described in detail in the Findings section. In essence, because all the websites selected for this study were 
reasonably large (indeed, a criterion for selection), they also tended to be complex. A criterion that re- 
quires the assessor to respond either yes or no to whether there is a feedback button becomes something of 
an interpretation. Does this mean on every page? On the homepage? Is once enough to satisfy the spirit 
of the criterion or is something more frequent intended? Such quandaries emerged more often than the re- 
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searchers had originally anticipated. The issue becomes one of comprehensive assessment of all the site’s 
webpages versus devising a valid sampling technique which may well miss key features of a website. 

For the situated assessments, the researchers used small sample sizes. Two to three assessors 
evaluated any given website. While this may be sufficient to establish proof of concept, the mean values of 
such small samples reported in the User Based Findings should be considered as indicators, not judgments. 
A systematic replication of the situated assessment methodology tested here but with more assessors — 
something that should be well within the capabilities of practically any federal agency — would enhance 
the precision of the evaluation. 

Other limitations of the study included time and money. For the funds available the research 
agenda was very ambitious. Multimethod data collection produced a sizable volume of data to be ana- 
lyzed. The time required to thoroughly digest all the data was longer than the time available. In both time 
and expense the researchers strove to maximize results. 

One of the stated aims of the study was to identify benchmark federal websites. It quickly became 
apparent that this hope would go unfulfilled. To accomplish such a task requires assessment of many more 
than four websites. That is not to say that the methodology presented here cannot be adapted to perform 
such a task, but rather that the task itself would be quite a different undertaking. 

Significance of the Study 

It is useful to collect experiences of specific government website developers which may contribute 
to knowledge that can be more generally applied. Mistakes costly in terms of time and dollars might be 
avoided. Technical knowledge can enable other agencies to “leapfrog” certain hurdles. Federal webmas- 
ters can use the data collection techniques and instruments developed and tested in this project can be used 
to improve their websites. The inclusion of user-based assessments is essential to developing high quality 
and useful websites. Thus, this research assists federal officials’ deployment and use of Web-based infor- 
mation resources and services. But by the same token, the critical success factors discovered through this 
study may be transferable to website designers in other settings. 
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Chapter 2: Policy Context 



Rapid proliferation of federal government websites is outpacing federal information management 
policies and dissemination guidelines that address issues associated with the dissemination of printed pub- 
lications. While many of the values reflected by these policies are still appropriate, /.<?., ensuring fair and 
equitable access to information for all citizens, and protecting information that may be sensitive or violate 
individual privacy, specific policies may not be practical for governing electronic information dissemina- 
tion via the Web. 

The federal government’s rapid transition to Web-based information dissemination raises the follow- 
ing fundamental questions: 

• Are federal websites being operated in a manner consistent with existing federal information 
policies? 

• Are new policies needed, or should federal information policies be updated to more realistically 
reflect the capabilities of this new medium? If so, in what areas are new or updated policies most 
urgently required? 

• Are federal agencies effectively employing the Web as an information dissemination channel? 

Federal websites are capable of providing fast, cost-effective access to an abundance of government 

information stored in a variety of electronic formats. Yet to enhance the overall usefulness and impact of 
federal websites, careful consideration should be given to their purpose, structure, and operation. It is time 
to reexamine key federal information policies in light of the pending impact of this potent new medium. 
This report presents results addressing the above questions through assessment criteria and methods for 
evaluating federal websites. 

Overview of Existing Federal Information Dissemination Policies 
Although websites provide a new medium for disseminating government information, OMB 
clearly indicates that federal websites fall under the purview of existing policies governing federal infor- 
mation dissemination activities (OMB, November, 1996). These policies encompass a variety of informa- 
tion-related issues: e.g., the types of government information that should be retained for storage in the 
National Archives; what should be distributed through the Federal Depository Library Program; whether 
and how the public should be charged for government information; and measures that should be taken to 
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safeguard and ensure the accuracy of information. The determination that federal websites must be oper- 
ated in compliance with these policies portends consequences for federal website administrators. The fol- 
lowing section discusses selected aspects of key policy instruments. 

The Paperwork Reduction Act 

As with many other policy issues, federal agencies must balance competing policy directives as 

they develop information dissemination programs. The PRA seeks to: 

Ensure the greatest possible public benefit from and maximize the utility of information created, 
collected, maintained, used, shared and disseminated by or for the federal government.... [and] 
provide for the dissemination of public information on a timely basis, on equitable terms, and in a 
manner that promotes the utility of the information to the public and makes effective use of in- 
formation technology ( Title 44 U.S.C. 3501 (2) and 3501 (7)). 

On the other hand, 44 U.S.C. 1 108 cautions federal agencies to make certain that information dissemination 
products are necessary for proper performance of agency functions or, as indicated by the PRA, “minimize 
the cost to the federal government of the creation, collection, maintenance, use, dissemination, and dispo- 
sition of information” (Title 44 U.S.C. 3501 (5)). In short, federal agencies are to maximize the usefulness 
of their information to the public - presumably by giving the public greater access to it — while minimiz- 
ing costs associated with doing so. 



OMB Circular A-130 

Written before the Web rose to prominence, OMB Circular A-130 is the de facto government- 
wide policy guidance for federal agencies using the Web as an information dissemination medium. In 
addition to directing agencies to maximize the usefulness of government information while minimizing 
dissemination costs, OMB Circular A-l 30 instructs agencies to: 

• Disseminate information products in an equitable and timely manner (Section 8.a.(5)(d)ii); 

• Take advantage of multiple dissemination channels (Section 8.a.(5)(d)iii); 

• Help the public locate the information, maintaining inventories of information dissemination 
products and developing search aids such as indexes and catalogs (Section 8.a.(5)(d)iv and Sec- 
tion 8.a.(6)(c) & (d) and OMB Bulletin 95-01); 

• Make certain that members of the public with disabilities have a reasonable ability to access the 
information (Section 8.a.(6)(f)); 

• Coordinate with members of the public, and state and local governments to ensure that informa- 
tion products meet their requirements, and provide notice when substantially modifying or termi- 
nating existing information products (Section 8.a.(6)(i) & Q)); 

• Make government publications, including electronic information dissemination products, available 
to the GPO for distribution through the library depository program (Section 8.a.(6)(g) & (h)); 

• Avoid imposing improperly restrictive practices on subsequent uses of information by others — 
particularly with respect to establishing exclusive distribution arrangements, charging fees or 
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royalties on subsequent re-dissemination of information, and charging user charges in excess of 
actual dissemination costs (Section 8.a.(7)), and; 

• Establish information protection procedures commensurate with the risk and magnitude of harm 
that would result from loss, misuse, or unauthorized access to or modification of such information 
(Section 8. a. (9)). 

OMB has revised Appendix IV of OMB Bulletin A-130 so that it now addresses the underlying 
rationale for many of the policies listed above, and clarifies changes from previous versions relevant to 
Web-based federal information dissemination activities. For example, Appendix IV clarifies the definition 
of "information dissemination product" to include both information products and services to ensure that 
policies not be interpreted differently based on the manner in which the information is delivered. Of inter- 
est to federal website administrators, Appendix IV includes an extended discussion of the Federal Deposi- 
tory Library Program stating OMB's position that: "electronic dissemination products generally should be 
provided to the depository libraries” {Federal Register 34, February 20, 1996: 6448). The manner in 
which electronic products should be provided to depository libraries is not identified. However, if websites 
evolve into primary dissemination vehicles and even replace some traditional functions of the depository 
libraries, the stipulation takes on importance to website administration. 

Electronic Freedom of Information Act 

The Electronic Freedom of Information Act Amendments of 1996 (Electronic FOIA; Title 5 
U.S.C. 552) propose to improve public access to federal information by directing agencies to make more of 
their information available in electronic form and to make reasonable efforts to provide records in the for- 
mat requested. Among other things, the Act requires agencies to: 

• Publish via computer telecommunications or other electronic means all information required to be 
published in the Federal Register, 

• Make available a list of statutes relied on by an agency to withhold information; 

• Make available for public access via electronic means agency materials previously available only 
through inspection and photocopying, and include in such materials specified information on ma- 
jor information systems; 

• Provide requesters with records in the form or format in which they are maintained; 

• Receive one-half of the fees collected from FOIA in order to offset compliance costs; 

• Process requests on a first-in, first-out basis; 

• Indicate where information has been deleted on the released portion of the record at the place 
where the deletions were made. 



While the Electronic FOIA did not explicitly reference the Web, the direction for agencies to publish via 
computer telecommunications certainly appears to encourage federal agencies to use the Web to meet these 
requirements. 

OMB Draft Guidelines for Agency Use of the World-Wide Web 
In recognition of the increasing use of the Web by federal agencies and the implications of exist- 
ing federal information dissemination policies on such practices, OMB developed and circulated draft pol- 
icy guidelines for federal agency use of the Web as an information dissemination medium (OMB, July 16 
and November 17, 1996). Although OMB was careful to point out that the guidance did not impose new 
requirements upon agencies, the memoranda did generate concern among many federal Web administra- 
tors. 



The initial draft (July 16, 1996) raised considerable concern among agency Web administrators, 
some of whom questioned the practicality and the cost-effectiveness of applying existing federal laws to 
the Web. At least one agency official indicated that if OMB guidelines are issued as official policy, some 
“agencies would stop using websites because of the burdens imposed by strict interpretation of the Federal 
Records and Paperwork Reduction Acts” (Harrefeld, 1996: 1, 52). Such reactions by agency Web adminis- 
trators highlight the significance of emerging policy and the possibility that tradeoffs in policy objectives 
may be required to optimize federal Web use. 

The draft policies provided in this guideline provide a useful starting point for assessing federal 
information-policy implications of agency use of the Web. Key provisions of the draft memorandum are 
summarized below. According to one or both drafts: 

• Agencies shall use appropriate management controls to provide reasonable assurance that infor- 
mation posted to their website is accurate, relevant, timely, and complete. The OMB emphasizes 
diligence but does not insist on absolute accuracy, suggesting a statement be included on the 
website reflecting when the information was last updated and reviewed for accuracy. This provi- 
sion was omitted from the November 17 draft. 

• An agency's website primarily supports the public affairs function of that agency and, as such, 
should be developed with access of the general public in mind. 

• Agencies should inform visitors to their websites whether they can expect to receive responses to 
comments submitted on their website. This provision was omitted from the November 17 draft. 

• Agencies should examine the appropriateness of the "linkages" between their websites and other 
websites with regard to sound public policy. For example, linkages to other government websites 





would generally be considered appropriate while linkages to commercial sites would typically be 
considered inappropriate. This provision was omitted from the November 17 draft. 

• Agency websites shall support only information directly related to the agency's mission. The 
agency website should not support personal information on agency personnel. 

• Agencies should track the costs associated with maintaining their website and assess the effect of 
electronic publication on their traditional dissemination programs. This provision was omitted 
from the November 17 draft. 

• Agencies shall not intentionally collect and maintain electronic mail addresses unless public notice 
of such collection is made and the information is maintained and used in accordance with a Pri- 
vacy Act systems of record notice. The November 17 revision specifically classifies email ad- 
dresses as “personal identifiers.” Databases which allow a user to find a record via a personal 
identifier must submit to Privacy Act oversight. OMB advises that it is the intent to search the 
data with a personal identifier rather than the capability of searching the data that determines 
whether the data should fall under the Privacy Act. 

• Regarding the coincidental collection of email addresses, such as might be accumulated while 
downloading comments from a discussion list that include an email address in the message header, 
agencies are instructed to treat these equivalently to routine paper correspondence and ensure that 
the addresses not be compiled into a searchable database. 

• Email addresses maintained by a software program for the purpose of managing electronic dis- 
cussion groups or mailing lists do not fall under the Privacy Act as long as agency personnel re- 
frain from accessing or manipulating the email address lists or using the email address list for 
other purposes. 

• Internet Protocol (IP) addresses are not personal identifiers and may be accumulated and manipu- 
lated in a database. The draft would require, however, that websites collecting users’ IP addresses 
post “conspicuous” notice of such collection and its intended purposes. 

• Agencies should provide an alternate means of access to information posted on their website. 

• websites shall include location aids in compliance with Government Information Locator Service 
(GILS) policy and standards. 

• Agencies must determine which information posted on their website constitutes an agency record, 
and if the information does constitute an official government record, treat the information in ac- 
cordance with statutory requirements. 

• Agencies should use their websites to augment their information dissemination — not to replace an 
existing means of disseminating information. Record copies are to be maintained in an appropri- 
ate format for access and preservation. 

• Website data collection efforts such as online surveys or questionnaires fall under the purview of 
the PRA and must be published in the Federal Register , approved by the OMB director, and as- 
signed a control number to be displayed on the collection instrument. The second draft clarifies 
that website suggestion boxes and non-specific requests for comment are exempt. 

• Agencies should treat information disclosures consistently. That is, if information is posted on an 
agency website, agencies are obliged to honor FOIA requests for the same or similar information. 
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• Agencies need to explore the use of additional controls to mitigate risks associated with permitting 
public access to federal systems. 

The November 17, 1996 OMB draft provides a fair and comprehensive interpretation of existing federal in- 
formation policies as they pertain to the operation and use of federal websites. Policy issues affecting the 
design and administration of federal websites are more fully discussed in Eschenfelder, et. al. (1997). 

A recent OMB memorandum to federal Chief Information Officers articulated five principles of 
website planning and operation (Katzen, April 14, 1997). 

1. Strategy and planning: Set clear strategic goals with performance measures demonstrating value; 
account for and maximize efficient use of agency resources; prepare for consistent service, 
maintenance and product delivery; and provide for the security of agency information systems. 

2. Service delivery: Attend to the needs of all Americans, including the financially disadvantaged, 
those with disabilities, and those without reasonable access to advanced technologies; partner 
with other agencies and organizations to enhance customer service; deliver services in timely and 
equitable terms without neglecting traditional delivery methods. 

3. Information access and dissemination: Strive for accuracy, relevance, timeliness and complete- 
ness of agency information made available electronically; provide cost-free access when practi- 
cable, but when fees are necessary limit them to the direct cost of dissemination; avoid 
improperly restrictive access practices; communicate regularly with the public in shaping and 
altering information services and products; support the Government Information Locator Service 
and disseminate available products through the Federal Depository Libraries. 

4. Collecting information from the public: Collect from the public only information necessary for 
the performance of official functions; solicit public comment and customer feedback; respect 
and guard the public’s privacy and ensure the security of their information; publish in the Federal 
Register agency activities which involve the collection and use (for system performance meas- 
urements or security) of email or Internet Protocol addresses, user statistics and activity logs, and 
similar data produced as a result of electronic interaction with the agency. 

5. Management of federal records: Agencies should create, use, preserve and dispose of electroni- 
cally published records in a manner consistent with the management of individual underlying re- 
cords, recognizing that in most instances compilations of existing records are not considered by 
the National Archives as new and distinct records. Agencies should use, preserve, and dispose of 
information generated through interactive websites in a manner consistent with the NARA guid- 
ance for email and other electronic records. 

The five principles outlined in the OMB statement are still under discussion, but range across a 
large number of issues. Principle number five addresses how to approach federal interactive websites 
(meaning those that deliver services or collect information via online application forms, surveys, etc.) from 
a records management standpoint. As stated above, current email and electronic records standards apply to 
websites. When the “interaction” between user and website results in a static document such as email or 
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forms, existing standards can be applied. However, research by McClure and associates reveals how 
problematic the issue becomes when the website as a whole is considered as an official information entity 
(McClure and Sprehe, 1996). 

Summary of Policy Issues 

Federal websites have an opportunity to enhance public access to public information. Despite the 
positive role the Web can play in information dissemination, tradeoffs are likely to be involved. Policy- 
makers and website administrators should consider a range of issues such as the possibility of unauthorized 
modifications to government information. Are websites poised to replace print records, and if so, will 
eliminating paper publications and disseminating information via the Web promote a technological de- 
pendency which leaves out significant portions of society? What measures can be taken to ensure that the 
privacy rights of U.S. citizens and residents are protected? 

Usually, resources of ail varieties tend to be limited, making full compliance with strict interpre- 
tations of the existing policies to be impractical, and often counterproductive. The potential implications of 
electronic information policies need to be better understood. If it appears that resource-strapped agencies 
will avoid posting information on the Web in order to avoid burdensome administrative requirements, 
public interests will be poorly served. There may be more effective means to address the issues raised by 
these policies without sacrificing full public access to public information. Promising new media such as 
the Web should be utilized as tools to make access greater and information better. The methodology de- 
scribed in Chapter 3 is a step toward enabling the convergence of information access and quality. 



Chapter 3: Methodology 

The research reported is exploratory in nature, and employs a combination of qualitative and 
quantitative techniques, including content analysis and descriptive measures for each website reviewed 
(Miles & Huberman, 1994; Creswell, 1994). Figure 1 summarizes the research methodology. The re- 
searchers assessed four federal (Executive Branch) websites using a variety of analytical techniques, as 
outlined below. Additionally, researchers conducted a site visit to one of the four selected websites. The 
site visit provided to the researchers Webserver log file data, email reference statistics, and other back- 
ground materials. 



Federal Website Selection 

To select a reasonably representative sample of federal webpages for assessment, the researchers 
established four basic conditions: 

1. The website must have been in existence for a minimum of six months. Six months provides time 
for the customary startup glitches and inconsistencies to be identified and remedied. Put another way, 
the website has had time to “mature.” If after six months problems persist, that may be an indication 
of more systemic problems. 

2. The website must have a person with designated technical responsibility for its maintenance. It 
became clear from numerous conversations between the researchers and various agency personnel 
that in many cases the website originated from the efforts of one or a few technical staff who knew 
some HTML coding and who perceived Web presence to be a benefit to the organization and/or to its 
constituencies. 

3. Use statistics are being recorded. This means that Web server log files are maintained that may be 
used to determine usage rates, problem areas, and aggregate user profiles at the organizational domain 
level, i.e.\ .gov, .edu, .com, etc. The logs help to identify particular webpages which have high error 
rates or other problems calling for closer inspection. The maintenance of use statistics is a selection 
criteria that applied only to the website participating in the site visit (Stat-USA). 

4. The website must have significant content. This does not mean simply a substantial amount, but 
rather, some depth and complexity. One of the consequences of hypertext is that it promotes com- 
plexity. Not only may inlinks direct attention around the architecture of a website, but outlinks incor- 
porate external websites or webpages into its content. For purposes of website selection, only the 
internal depth and complexity of websites are considered. For assessment, appropriateness of out- 
links are considered, as well. 

Using the above selection criteria, the study team compiled a candidate list of nine federal websites. Mem- 
bers of the study team addressed several graduate-level classes at Syracuse University School of Informa- 
tion Studies to recruit assessors to fill the situated proxies role for the corresponding assessments. 

Appendix 2B contains the original list of federal websites presented to the students, as well as summary 





Figure 1 
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information about those sites. 



It was crucial for the integrity of the situated methodology, as well as the validity of the approach 
used in this research that the assessors be interested users of the federal websites they evaluated. Because a 
primary aim of this study was to explore ways that federal Web administrators could employ user feedback 
in the overall assessment of their websites, the user proxies made the final selection of websites to evaluate 
from a list of qualified sites. The use of the term “proxies” for situated assessors should not be taken to 
mean that they are unrepresentative of “normal” federal website users. The user demographic of federal 
websites reflects the heterogeneity of the hundreds of websites themselves. Therefore, the situated users 
functioned as proxies in that they tested the assessment methodology. As will be discussed later in this re- 
port, the proxies felt that their experiences evaluating the websites served to validate the criteria-based as- 
sessments as a proof of concept. 

The four federal websites selected by the situated users varied broadly in size, complexity, and 
mission. These sites were: 

• The National Institutes of Health, which deals with a range of public health topics. 

• The National Archives and Records Administration, which is responsible for federal information 
resources management. 

• The Census Bureau, which has a large website operating under the auspices of the U.S. Depart- 
ment of Commerce. 

• Stat-USA which, like the Census Bureau, is attached to the U.S. Department of Commerce. Stat- 
USA provides U.S. statistical data to its website’s visitors. 

Stat-USA administrators agreed to host a site visit by the researchers, and became the case study for the 

combined multimethod assessments. Stat-USA maintains a fee-based website subscription service which 

managed to recover its operating expenses during the second year of the website’s operation. 

Development of Evaluation Protocols for Situated, Technical, and Policy Assessments 

The study team developed and pretested three sets of questionnaires containing scripted assess- 
ment criteria. These efforts drew on previous work by McClure and Lopata (1996), and Bertot and 
McClure (1996). The first set of scripted criteria applied to the situated proxies’ assessments, the second to 
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technical elements of websites. The third dealt with the conformance of the websites to policy guidelines 
about federal information dissemination, and particularly to website development. 

The study team developed the assessment criteria from four sources: 

1 . A similar website assessment study which focused on the Maryland State Library Sailor Network 
(Bertot & McClure, 1996) 

2. A literature review conducted by members of the research team, including both bibliographic 
sources and online searching 

3. An intermediate version of website evaluation criteria previously developed by the study team 
(Eschenfelder, Beachboard, McClure & Wyman, 1997) 

4. Comments collected from a focus group held in October 1996 of experienced website developers. 
These sources provided a survey of existing assessment criteria and approaches, and methods for selecting 
and refining them into more effective forms. 

Prior to conducting evaluations, the study team collected assessment criteria through a literature 
review. Team members separated the criteria into groups according to the types of websites to which they 
applied. For example, many existing assessment criteria are directed at raising a website’s visibility for 
commercial purposes. Many of these were not applicable to federal websites were disregarded for this 
study. Another category which attracts evaluators is the aesthetic sophistication of a site. This last crite- 
rion is not important to this study. 

In developing a survey instrument, the study team initially decided to adapt the Sailor study ques- 
tionnaire, cited above. In reviewing the Sailor instrument and that study’s Final Report , the team con- 
cluded that the Sailor instrument, as written, was too lengthy and broad in scope for the purposes of the 
present study. Having narrowed the scope of the instrument, the study team held a focus group of six ex- 
perienced website designers and to review some of the proposed questions and seek suggestions for new 
questions. Study team members circulated a draft of the assessment criteria to experienced website devel- 
opers from the School of Information Studies’ graduate computer lab. The input from these experts helped 
to identify problematic or ambiguous criteria. After assessing websites of their own choosing with a proto- 
type of the scripted criteria the subjects participated in a debriefing session. This focus group provided a 
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forum for recommendations to the study team for refinements to the criteria. Some of the key issues raised 
by the group included: 

• Will users asked to assess a website with splash pages incorrectly answer questions about the 
agency “homepage”? Splashpages set the mood or site metaphor, but contain very little informa- 
tion. When referring to an agency “homepage,” one usually thinks of the first page that appears 
when a user accesses a site. Some styles of homepages however, use “splash pages” as a first 
page. 

• To what extent can a user easily distinguish between a plain graphic and a graphic which serves as 
a link? 

• To what extent is navigational assistance for a page unnecessary, if good page designs do not re- 
quire navigational assistance information? 

• How can good survey questions be developed with no knowledge of the agency site’s primary us- 
ers or target audience? 

The study team combined the technical and policy criteria into a single assessment session. The 
reason for this was pragmatic: The first task was believed to require neither user feedback nor expertise, 
and so in principle could be accomplished by anyone. The policy assessment, on the other hand, required 
special skills. The policy assessors doubled as the technical assessors to conserve time and expense. The 
assessors answered the technical criteria after exploring each of the four websites approximately one hour 
apiece. Policy assessments took somewhat longer. 

Development of Think Aloud Protocol for Use in Situated Assessments 
The researchers pretested the situated assessment protocol that was based on the “Think Aloud” 
procedure, which as described by van Someren, Barnard and Sandberg, “consists of asking people to think 
aloud while solving a problem and analyzing the resulting verbal protocols” (van Someren, Barnard & 
Sandberg, 1994: xi; also Shapiro, 1994: 1-14). Because websites are dynamic by nature, the researchers 
determined that videotaping the monitor display during the assessors’ searches would preserve the specific 
context of their searches, retain relevance to their commentary, and capture their verbal comments. 
Videotaping did not occur until the situated proxies assessments. 

Using the information and suggestions from the Webmeisters, the study team revised the scripted 
criteria for the situated and technical assessments. See Appendix 2C for the linkages between user ques- 
tions and selected assessment criteria. Other survey instruments included demographics questionnaires and 
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focus group probes for the situated users and the site visit participants. Appendices 2F and 21 provide re- 
productions of the instruments. 

The study team administered a second pretest later to three additional graduate students specializ- 
ing in information retrieval. Except for no videotaping, the second pretest sessions followed exactly the 
procedure to be used with the situated proxies. The purpose of this additional step was to detect any re- 
maining glitches in the overall approach. During this exercise the study team noted that as the assessors 
became involved in the websites they forget assessment criteria. As a result of the second pretest, for the 
full assessment the researchers placed a small banner (Figure 2) at the top of the monitor of the computer 
that the assessors used as a mnemonic for the categories to be addressed. 



Figure 2: Mnemonics for situated assessors 



• Navigation 


• Quality 


• Usefulness 


• Presentation 


Buttons 


Unbiased 


Good Organization 


Useful Headers 


Ease of Moving Around 


Fulfills Needs 


Clear Scope 


Good Graphics 



The study team intended the banner to focus the assessors’ comments on the selected assessment criteria. 
Situated assessors viewed the assessment criteria for a few minutes prior to conducting their online ses- 
sions, so that as they conducted their sessions they would be cognizant of the types of criteria they would 
address at their session’s conclusion. 

The situated proxies verbalized their thoughts/reactions as they moved through the websites. Im- 
mediately after finishing their Think Aloud assessment of the selected site, the situated proxies filled out a 
form containing assessment criteria. The study team analyzed the resulting sessions according to basic 
content analysis techniques (Weber, 1990). All but one of the situated assessors participated in a focus 
group within a week of the online sessions. Suggestions made during the focus group led to refinements in 
the assessment instruments, to the development of probes for the site visit, and to an 1 1 -minute videotape 
composed of portions of three assessment conducted on the site visit website. Researchers played the 
videotape during the site visit. 
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Website Assessment Techniques 



The techniques developed by the researchers to assess federal websites divide into four general 
categories: System-based, user-based, design-based, and developer-based. In the original Research Pro- 
posal six data collection activities were outlined. Of those six, the researchers dropped online popups for 
reasons described below. However, it became possible during the course of the research to add Web server 
log file analysis to the project. While popups offered a means to collect user feedback to elements of the 
selected websites, they would have been fairly unobtrusive and limited to analysis within the User Based 
general category. Log file analysis, on the other hand, while largely system based, also provides a means 
of observing traffic through a website, and is therefore arguably a user based option as well. In fact, sev- 
eral of the sub-categories outlined below interconnect and inform the others. Due to this fact, the research 
has an inherently iterative character: As the research unfolded, lessons learned during the performance of 
one task led to refinements in the instruments and methodology of that task, and frequently to others. 
System-Based Website Assessment 

Website Mapping 

Objectives: 

• Evaluate a sample of federal websites 

• Offer recommendations to increase the usefulness of federal websites to libraries and other 
users 

• Indicate analytical tools with which federal website administrators can analyze site structure and 
use. 

The study team used a website management application, WebMapper™, by NetCarta, to obtain a 
structural overview of each site . 3 This commercial package detects obsolete hyperlinks, indicates whether 
links were internal to the site or directed the user to external websites. The WebMapper application detects 
duplicate pages, in addition to giving counts and locations of all links, images, applications and media 
within, and linking to, a website. Objects such as video, Internet applications (e.g., .pdf files generated by 
Adobe Acrobat), sound, and gateways get identified and assigned icons, which may in turn, be summarized 



3 After reviewing several off-the-shelf applications, the study team selected NetCarta’s WebMapper™ to accomplish this 
task. Microsoft Corp. recently acquired NetCarta. Product details may be reviewed at <http://www.microsoft.com/ 
corpinfo/press/1996/dec96/netcarpr.htm>. 



Figure 3: Website Assessment Categories 



in specific reports. WebMapper detects a variety 
of structural errors (including 404 errors “file not 
found”). Since WebMapper can survey a site 
without special permissions (one study website 
had numerous passworded pages which mapped 
without any problems), the researchers attempted 
to map all four federal websites selected for this 
study. 

Mapping begins at the selected site’s 
homepage, and then proceeds to identify and map 
descending levels until it reaches a point where it 
finds no further links or objects. One of the most 
efficient means to map a site is to download the 
structure into a local cache. To do so can be 
memory intensive. One of WebMapper’s most 
useful features is its ability to compare a website 
structure at two points in time. This option 
permits the Web administrator to review all 
changes made to a site from one mapping to the 
next. The researchers did not include a longitu- 
dinal component in this study, though. 

Website mapping is a system-based data 
collection activity involving no user feedback. 

Webmapping is a data collection activity that 
applies to all four websites assessed in this study. Although the Webmapping application does not provide 
any user data for the website, in concert with server logs a powerful analytical combination results. The 



I. System-Based 

A. Website mapping (all websites) 

B. Web server log file analysis (Stat- 

USA) 

II. User-Based 

A. Situated proxies assessments (all 
websites) 

1 . Website selection 

2. Pretest 

a. 1ST Webmasters 

b. 1ST graduate students 

3. Online assessment instruments 

a. Researchers’ script 

b. Demographic form 

c. Assessment criteria (used) 

d. Assessment criteria (revised) 

4. Focus group material 

a. Probes 

b. Transcript 

B. Online popup questionnaires 

(dropped from study) 

III. Design-Based 

A. Technical assessment (all websites) 

B. Policy assessments (all websites) 

IV. Developer-Based 

Site Visit (Stat-USA) 

1. Interviews 

2. Focus group 

3. Probes 

4. Demographic form 

5. Videotape showing 
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present study compares the output of the Site Summary Report which gives a detailed overview of an entire 
site: 

• Object Statistics: The number and size (where relevant) of pages, images, gateways, major mime 
types, and other resources. 

• Status Summary: The counts and error status of onsite and offsite objects (pages and other re- 
sources) and links. 

• Map Statistics: Information about the date and time your site was mapped, the number of levels 
mapped, and the average number of links per page. 

• Server Summary: Information about a site's server. 

The Webmapping application produced reports providing indicators of pages that may be especially com- 
plex or central to the overall website. This is an unobtrusive data collection technique that did not require 
participation of website administrators of the mapped sites. 

Web Server Log File Analysis 

Objectives: 

• Evaluate a sample of federal websites 

• Offer recommendations to increase the usefulness of federal websites to libraries and other users 

• Indicate analytical tools with which federal website administrators can analyze site structure and 
use. 

The study team prepared methods to assess four types of log files: access, agent, error, and refer- 
rer (Bertot, McClure, Moen, & Rubin, 1997; Stout, 1997; Noonan, 1996). 4 One federal agency — Stat- 
USA — agreed to permit the researchers to analyze its Webserver log files (Rubin, 1996). One type of log 
file, the error_log, for instance, allows the detection of webpages or hyperlinks that may pose problems to 
the user. Examples could include slow page loads, or defunct links. The files which the study team ulti- 
mately received included only accessjogs for an eight-day period. Even limited to the access_logs, how- 
ever, useful information about website use came to light. Details can be found in the Findings section. A 
study team member designed software for statistical analysis of the log files using custom PERL and CGI 
scripts (Rubin, 1996; Bertot and Rubin, 1997). Details of this portion of the research effort are available 



4 Descriptions of what can be accomplished through log file analysis is currently available at 
<http://istWeb.syr.edu/stats.html>. See also Rubin, 1996; Bertot, McClure, Moen, & Rubin., 1997; Stout, 1997. 
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through the Web at the OCLC Project webpage under the title: “World-Wide Web Access Statistics for 
istWeb/ syr.edu.” <http://istWeb.syr.edu/Project/Faculty/fedweb.html> 

There are four variables in the accessjog which can be used for analysis: 

1. Domain name or Internet Protocol (IP) number. Here is a sample line of text from an ac- 
cess_log: smx-ca8-50.ix.netcom.com - - [30/Sep/l 996:02:57:07 -0400] "GET /Proj/m ain.html 

In this example we know that the user’s computer had the following domain name: smx-ca8- 

50.ix.netcom.com. In the United States six suffixes are used to describe an organization type (though re- 
cent developments indicate an impending expansion of the present nomenclature) 5 : 

• .com - US commercial business ( e.g ., microsoft.com, sony.com) 

• .gov - US governmental agencies (e.g., whitehouse.gov, epa.gov) 

• .edu - US Educational (colleges, universities) (e.g., syr.edu, bu.edu) 

• .net - US network providers (e.g., intemic.net, intemet.net) 

• .org - US non profit organizations (e.g., madd.org, greenpeace.org) 

• .mil - US military (e.g., army.mil, navy.mil) 

The user in the example connected to the Web server from a commercial network. The domain can be 
broken down further to the sub-domain; for example, how many times microsoft.com, ibm.com or us.mil 
connected to the website. It can also useful to know how many hits the server is getting from countries 
outside the United States. 

2. Date and Time. Using the same example, we know that the user accessed a page on September 
30, 1996 at 2:57:07 a.m. on a twenty-four-hour clock. 

3. Item accessed. The word “item” can mean an image, movie, sound, or HTML file. The above 
example indicates that the visitor accessed the page main.html. Note that the full path name (from the 
document root) is given to avoid confusion, (e.g., there may be more then one main.html on a server). The 
“item accessed” statistic can show a website administrator the most or least successful parts of a site. 



5 During preparation of this Report the International Ad Hoc Committee on Top Level Domain (Internet) Names issued a 
Memorandum of Understanding proposing the following seven additions to the current IP domain types: 

.firm for businesses, or firms 

.store for businesses offering goods to purchase 

.web for entities emphasizing activities related to the WWW 

.arts for entities emphasizing cultural and entertainment activities 

.rec for entities emphasizing recreation/entertainment activities 

.info for entities providing information services 

.nom for those wishing individual or personal nomenclature 



23 



33 




4. Threading of Users. Understanding the path (thread) through which a user navigates a site can 
unfold multiple uses, such as: 

• Entrance: The percentage of users who enter a site at the homepage, as opposed to directly ac- 
cessing a page within it. This can help determine where to put an announcement ( e.g ., new con- 
tent). If a Web administrator puts an announcement only on the homepage a large number of 
users who bypass the homepage will never see the announcement. 

• Exit: The page(s) from which users most frequently leave a site. It could be that a large number 
of users leave a site on a specific page. This alerts a Web administrator to re-examine that page. 
Maybe a link is down, maybe a graphic is taking to long to load, maybe the content needs some 
work. 

Clock analysis: The amount of time users spending on pages. Although a full page of text which 
receives an average of 800 accesses a week might seem popular, if the average time a user looks 
at that page is a few seconds there may be problems with it. 

• Download time: How long a webpage takes to load. Users access sites with different bandwidth 
constraints (e.g., 9600, 14.400, 28.800 bps, ISDN, Ethernet). A page that takes six seconds to 
download for one user, may take six minutes for another. 

• Analysis of agent_log: Can tell what sort of browser the use to access the website. Netscape’s 
Navigator and Microsoft’s Internet Explorer are graphical browsers, but there are still people who 
depend on, and perhaps prefer, textual browsers such as Lynx. Agentjogs can also tell the 
website administrator which version of the browsers connect to the site, and which operating sys- 
tems the clients are using (e.g., Windows; 32 bit, Macintosh; PowerPC, SunOS; 5.3). 

Details which the access_log collects on clients connecting to a website can be very useful to the website 

designer and administrator in providing an environment which significantly enhances accessibility or, at 

least does not hinder users’ access to information. 

User-Based Website Assessment 



Situated Proxies Assessments 

Objectives: 

• Establish evaluative criteria for reviewing federal websites 

• Evaluate a sample of federal websites 

• Identify and define key issues regarding representation and access to information resources 
through federal websites 

• Offer recommendations to increase the usefulness of federal websites to libraries and other users. 
Situated proxies assessed the four selected federal websites and subsequently participated in a fo- 
cus group to discuss key issues and problems. Proxies assessed websites according to scripted criteria and 
then evaluated the criteria themselves. The proxies thus played a dual role in this study. At least two 
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proxies explored each of the four websites selected. The study team encouraged the situated proxies to re- 
spond as naturally as possible to their respective websites and to consider criteria other than those supplied 
by the researchers. Each session lasted approximately one hour and had the following structure: 

• Orientation: Description of Think Aloud technique, read through of criteria, assessors complete 
demographics form -- 10 minutes 

• Search session: Videotaped assessments — 20 - 40 minutes 

• Debriefing: User provided summary of overall encounter. User also could evaluate the criteria 
themselves and indicate any additional information they felt should be included — 20 minutes. 

Shortly after completion of all situated assessments, the proxies and researchers met collectively to discuss 
the search sessions and to further refine the criteria into a more concise, effective set. 

A website’s presentation, contextual ization, organization, and ease of use are all factors that affect 
every user’s ability to access and utilize website contents effectively. Assessing some of these characteris- 
tics entails subjective judgments. To test assessment criteria that address subjective elements of website 
design, the researchers recruited volunteers from graduate courses in Syracuse University’s School of In- 
formation Studies. Ten students participated. These students served as “proxies” of ordinary users of each 
of the four federal websites included in the study. To enhance a naturalistic approach, each student se- 
lected a website to assess from a list provided to them by the researchers. To their assessment sessions 
they each brought a specific question related to their expectation of what information that website was 
likely to contain, against which they could evaluate the features and effectiveness of the site. In this sense, 
the proxies may be considered “situated” within an information problem context (Dervin & Nilan, 1986). 
The situated assessment criteria divided into two main categories, with a “general” category for several 
criteria thought to be important but not well-suited to the major divisions. The main categories for situated 
assessment criteria were: A. Ease of use -- presentation and navigation; and B. Content — quality and use- 




fulness. 



Table 2. Situated Criteria for the Evaluation of Websites 



CONTENT 



;xEASE::;0F::iUSE 



A. Presentation Issues 

1 . To what extent are the text headers under- 

standable? 

2. To what extent do the headers accurately 

reflect the content they describe? 

3. Rate the overall site design 

4. To what degree was the information at the 

site located where indicated by the head- 
ers? 

5. To what extent is the design consistent 

throughout the site? 

6. To what extent is the use of graphics ap- 

propriate for the site? 

7. To what extent are any of the visual ele- 

ments distracting? 

B. Navigation Issues 

1 . The graphics helped me find my way 
around the site. 

2. It was easy to move to where I wanted to 
go. 

3. It is easy to understand what the navigation 
icons mean. 

4. It is easy to find what I want without using 
a search engine. 

5. Chance played a big part in my finding the 

information I wanted. 



C. General Questions 

1 . It would be easy for me to become skillful 
at using this site. 

2. The site was easy to use. 

3. Teaching a beginner to use this site would 

be easy. 

4. It was easy to provide feedback to this site. 



A. Quality Issues 

1. To what extent did the site present different 

types or sources of information about your 
topic, thereby avoiding bias? 

2. To what extent does the content fulfill your 

needs? 

3. To what extent does the site provide in-depth 

information? 

4. Rate the overall quality of the site 



5. I would recommend this site to a colleague. 

6. To what extent do you think the information 

provided by this site is true or correct? 

7. To what extent does information quality vary 

within the site? 

B. Usefulness Issues 

1. The content of the site matched my expecta- 

tions. 

2. The site clearly indicated its scope and con- 

tent. 

3. The site clearly described its services and in- 

formation. 

4. The site organized its information in a way 

that was easy for me to understand. 

5. Using this site would enable me to accomplish 

my desired tasks more quickly. 

6. The site provided the precise information I 

needed. 

7. The information on the site was relevant to 

me. 

8. I understand the purpose of this site. 

C. Open-ended Questions 

1 . Which aspect of the site did you like the 
most? 

2. Which aspect of the site did you like the least? 

3. Please make one suggestion for improving 
the website. 
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Online Popup Questionnaires 

Objectives: 

• Establish evaluative criteria for reviewing federal websites 

• Evaluate a sample of federal websites * 

• Identify and define key issues regarding representation and access to information resources 
through federal websites 

In concert with the agency participating in the case study, the researchers intended to install hy- 
perlinks from that website to two online popup questions chosen from situated proxies’ assessment. The 
popups would have served as a proof-of-concept for collecting online user feedback in response to tested 
criteria. The period of popup data was to be two weeks. As described earlier in this document however, 
website administrators concluded that the popup technique risked annoying frequent website users. Fur- 
thermore, technical problems emerged whose solutions exceeded the scope of the project. Consequently, it 
became necessary to eliminate the online popup questionnaire data collection from the study. 
Design-Based Website Assessment 

Technical Assessments 

Objectives: 

• Establish evaluative criteria for reviewing federal websites 

• Evaluate a sample of federal websites 

• Identify and define key issues regarding representation and access to information resources 
through federal websites 

The technical assessments considered elements of website design and operation. The assessment 
criteria divided into the following categories: A. Ease of Use issues — presentation, navigation and help; 
B. Content issues — quality, usefulness, and currency. Determining if the site includes a functioning user 
feedback feature or whether its hyperlinks work was not viewed as requiring much judgment. Therefore, 
the technical assessment was intended to be an objective assessment which asked how well the site func- 
tioned technically, regardless of content. A number of scripted criteria used in the technical assessment 
derived from previous research (Bertot and McClure, 1996). Others came from the literature review. Us- 
ing scripted criteria, three members of the study team assessed the four selected federal websites. Appen- 
dix 3A describes the criteria employed for the technical assessment. 




27 37 



Table 3: Technical Assessment Criteria 



Technical Criteria 


Census 


NIH 


NARA 


Stat-USA 


Content: Quality Issues 

1. Do hyperlink headings and descriptions match the informa- 
tion found through the links? 

Usefulness Issues 

1. Are descriptions given to further describe the headers? 

2. Are services offered on the site? 

3. Are the services functional? 

Currency Issues 

1 . Does the website include dates indicating when pages were 
last updated? 

2. How long since the last update? 

Ease of Use/Presentation Issues 

1. Are the banners consistently designed? 

2. Are the banners consistently placed? 

3. Are the navigation bars consistently designed? 

4. Are the navigation bars consistently placed? 

Navigation Issues 

1 . Do hyperlinks connect to gopher menus? 

2. Is any of the Site restricted by passworus.'' 

3. Is the page longer than three screens? 

4. Do pages longer than three screens provide anchors? 

5. Does the site consistently differentiate between graphics 
and graphical links? 

6. Does the site have a consistent way of indicating text and 

graphical hyperlinks? 

7. Do you need to use the browser to navigate? 

8. Does each page provide an indication of where you are in 
the site? 

9. Does each page indicate what site you are on? 

10. Does the site warn you when you take an outside link? 

Help Issues 

1 . Is there a person or address that can be reached to provide 
feed back or to ask questions? 

2. Are large documents broken up into subsections? 

3. Are indexes provided for large documents and subsections? 

4. Does the site provide a download option for large docu- 
ments? 

5. Does the site warn you about file sizes? 

6. Is a text only version of the site available? 
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Three members of the study team conducted the Technical Assessment. Most of the technical as- 
pects of the assessment evaluated objective website elements which did not require situated users to make 
evaluations. For example, it might be deemed important to know if a feedback mechanism of some sort 
exists on the website. This requires a simple observation to confirm, and therefore was considered an ob- 
jective judgment. 

Policy Assessment 

Objectives: 

• Establish evaluative criteria for reviewing federal websites 

• Evaluate a sample of federal websites 

• Identify and define key issues regarding representation and access to information resources through 
federal websites 

• Identify and analyze key information policy issues related to the design, development, and manage- 
ment of Web-based information resources and services. 

To conduct a policy assessment of the selected websites, the assessors relied on the literature re- 
view prepared for this study (Majchrzak, 1984: 92-100). The literature included draft policy documents, as 
well as other information on conditions and issues being considered by federal officials in regard to the 
World Wide Web. 6 The federal World Wide Web Consortium offers a good beginning at addressing some 
of the technical aspects of website development <www.dtic.mil/staff/cthoms/ guidelines>. Appendices 3C- 
D describes the criteria employed for the policy assessments. 

The researchers assessed the policy contexts of the four selected websites. The policy assessment 
proceeded from the fundamental question: How well do the websites conform to existing federal guide- 
lines for information dissemination? To answer that question, the researchers located and compared policy 
documents targeting either government-wide or agency-specific website development and maintenance. 



6 Guidelines for Agency Use of the Internet . . . (op. tit ), Office of Management and Budget; Use of the Internet. 
<Http://www.os.dhhs.gov> Department of Health and Human Services; Guidelines for Establishing and Maintaining a 
Department of Defense Web Information Service. <http://www.dtic.mil/defenselink/Webguide.html> Department of De- 
fense; World Wide Web NIH Guidance (1995), <http://www.nih.gov/od/oirm/ guideli2.html> National Institutes of 
Health; World Wide Web Home Page Guidelines. (May 6, 1995), <http://skydive.ncsa.uiuc.edu/consortium/guide 
/hmpggl.htm> NCSA/NSF World-Wide Web Federal Consortium; 'WWW Server Standards and Guidelines. 
<http://inet.ed.gov/ -kstubbs/ wwwstds.htmi> U.S. Department of Education; MSFC Guidelines for Administering 
World-Web Resources (1995) <http://www.msfc.nasa.gov/Webmasters/policy/ guidelines. html> Marshall Space Flight 
Center - National Aeronautics and Space Administration; DTIC WWW Sever Standards and Guidelines. (Final Draft : 
4/24/95). <http://www.dtic.dla.mil/staff/trefzger/standards.html> Defense Technical Information Center; Policy for 
Internet Information Dissemination Using the World Wide Web. (Draft: 8/9/94). <http://banjo/ cise-nsf.gov/cise/ 
ClSEWebPolicy.html> National Science Foundation. 
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Details of the policy assessment framework developed and applied by the researchers to the federal 
websites in the study can be reviewed in Eschenfelder, Beachboard, McClure and Wyman (1997). The re- 
searchers concluded that the policy assessment portion of the study required a degree of expertise difficult 
to operationalize in a user feedback study. For this reason, three research team members performed the 
policy assessment according to scripted criteria developed for the task. 

Table 4: Policy Assessment Criteria 



Policy Criteria 


Census 


NIH 


NARA 


Stat-USA 


1. Does the website provide an email link to a Web adminis- 
trator? 

2. Does the site offer users the opportunity to make comments 
about the site’s content or design? 

3. Is there indication that visitors should expect responses to 
their comments? 

4. Is there a notice of when the website or specific pages were 
last updated? 

5. Does content appear appropriate for public access? 

6. Does content appear appropriate for the agency’s mission? 

7. Does content appear to advocate political positions? 

8. Are there external hyperlinks to commercial sites on the 
website? 

9. How appropriate do links to commercial site seem? 

10. Are any data collection mechanisms included on the 
website? 

1 1. Do data collection mechanisms appear appropriate in 
terms of the agency’s mission? 

12. Does the website provide pointers to alternative means of 
accessing the content (telephone numbers for ordering 
paper copies)? 

13. Are the location aids GILS compliant? 











Developer-Based Website Assessment 



Site Visit 

The study team combined the research techniques in a site visit to one of the four selected federal 
websites: Stat-USA, operating from within the U.S. Department of Commerce. Stat-USA availed itself to 
the researchers as a case study (Yin, 1994). The site visit was composed of individual interviews with key 
personnel involved in website development and maintenance, and a focus group of those interviewed and 
others invited by the head of the Stat-USA operation. The researchers distributed probes to the focus group 
participants for discussion (see Appendix 21). Many of the probes derived from the situated assessors’ 
feedback and subsequent focus group. Also, federal participants filled out a demographics form which 
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included some open-ended questions regarding the Stat-USA’s experience and opinions regarding federal 
website design and operation (see Appendix 4B). During the focus group the researchers showed a video- 
tape of three situated assessors’ evaluations of the Stat-USA website and then discussed the contents of the 
tape (see Appendix 4C for a transcript of the videotape). 
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Chapter 4: Findings 



This chapter presents results of the data collection efforts. Each collection effort is presented in- 
dividually. The data collected during the present research is very heterogeneous, ranging from statistical 
measures of user log files and location and counts of website errors, to survey and interviews. Analysis 
between data collection categories requires resources that exceed the scope of this study, but in some cases, 
the researchers identified ways in which one set of findings correlates to another. 

A general result that emerged as the research progressed was the lack of a clear baseline from 
which to assess websites against one another. It is doubtful whether such an indexing might have merit, 
since the purposes, content and organization of federal websites can differ widely. Overall, the research 
findings support the methodological approach developed by the researchers for evaluating federal websites. 
However, one of the original expectations of the researchers was that the assessments could help to identify 
benchmark federal websites. At the least, many more websites will need to be assessed before a “best 
practices” standard can be set (Novak and Hoffman, 1996). Furthermore, websites are typically highly dy- 
namic. If one site is designated a benchmark, given certain contexts and conditions, if its administrators 
revise it the next day, does the designation stand? A better approach to benchmarking federal websites 
may be to follow-through with an individual assessments and rate the site against the evaluative criteria 
rather than compare it to other websites. Accordingly, since it was never an aim of the researchers to es- 
tablish a rating index (which might erroneously suggest an objective standard), the results reported in this 
chapter maintain a general character, with a tighter focus given to the Stat-USA case study. A principle 
objective of the researchers was to determine the validity of the methodological approach to assessment de- 
scribed in this study; and to that aim the analysis primarily applies. 

Website Mapping 

The following tables illustrate a range of important features of three of the four federal websites 
selected for the present study. At a glance, basic comparisons may be made between the depth and com- 
plexities of the three sites. In order to fairly assess the data presented below, the reader should bear in 
mind that major differences exist between the three websites in a variety of ways. Each organization’s 
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mission and functions are highly specific. Due to the differences in mandates, the information content 
varies a great deal from one site to the next. Stat-USA, is a fairly self-contained and autonomous entity 
which provides to its subscribers large databases; NARA offers much more archival materials; and the 
NIH website is comprised of an association of 24 separate institutes, centers and divisions making it or- 
ganizationally complex. Organizational imperatives tend to be reflected in the websites’ design and main- 
tenance, although Web mapping reveals subtleties in those selected images. Data organized into three 
tables appearing later in this chapter provided the researchers glimpses behind the curtain, into the struc- 
tural systems that undergird information presentation and use, but systems important to the overall acces- 
sibility of the websites. 

The Map Statistics data Table 5 is straightforward. The Stat-USA website directory structure ex- 
tends two levels deeper than either NARA or NIH. Stat-USA’s six levels is relatively deep, and may indi- 
cate to the Web administrator that the site is particularly complex, or that protracted searching for 
information may be commonplace at the website. Comparing the website map with log files can help de- 
termine the amount of effort required of users. On the other hand, one can see that the average number of 
hyperlinks for individual Stat-USA webpages is a very spare four. Both NARA and NIH are nearly four 
times as hyperlinked. As will be discussed more fully in the situated proxies Findings section, the users 
noted that the Stat-USA website seemed to be an HTML “shell” overlaying database files. Here we have 
possible verification of the users’ perceptions, thus reinforcing the validity of the combined methods. The 
matter may be analyzed further by reference to a WebMapper reporting function that maps links to pages 
(not shown on the summary tables). 

Because so many of Stat-USA’s clients download data directly into Excel, or other applications, 
the presence of HTML tags can reduces the usability of the data by acting as contaminants. Here is a 
situation where the developers’ and proxies’ perceptions differed. Stat-USA addressed the difference by 
suggesting that their website subscribers were quite different from the student proxies used for the situated 
assessments. This may be quite true. It demonstrates the caution with which assessors should proceed. 
Log file analysis can help to provide a stakeholder profile of present users. If an aim of the website is to 
increase use (as is the case with Stat-USA), the caution points in both directions: It could very well be the 
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case that situated proxies’ assessments can show the way to new users. Finally, the Map Statistics report 
describes which Web server software the site is running, as well as which HTTP version. 



Table 5: Map Statistics 





Stat-USA 


NARA 


NIH 






Map Statistics 




Levels 


6 


4 


4 


Avg. 

Links/Pg. 


4 


15 

Server Summary 


14 


Domain 


www.stat-usa.gov 


www.nara.gov 


www.nih.gov 


Server 

Version 


Netscape-Commerce/1 .2 


Netscape-Commerce/1 . 1 


Apache/ 1.1.1 


HTTP 

Version 


1.0 


1.0 


1.0 



Table 6 below presents a more detailed picture of the websites. WebMapper refers to individual 
pages, images, gateways, audio or video files, Mailto, applications or other website resources as “objects.” 
From Table 5 one can see Stat-USA is by far the largest of the three websites, containing 912 objects total, 
of which 420 are webpages. The storage capacity required for the website itself exceeds 34 megabytes. 
Also, the Stat-USA site far outdistances NARA andNlH in the number of applications running on its 
server, with 66 total, compared to 2 for NARA and none for NIH. It is also easy to see that despite having 
a total memory requirement of less than 300 kilobytes, NARA has substantially more images loaded on its 
website as either of the others. Even that is not especially graphics intensive for a website with 261 pages 
(the graphics files - 84 - are counted separately from pages). NARA and NIH are quite similar in size and 
depth, as well as in the type and number of gateways, and applications running or not present on their sites. 
That Stat-USA with a far larger site has so many fewer graphics highlights the intensive database nature of 
the Commerce website. The applications objects for Stat-USA account for 30 of its 34 megabytes. The 
fact that WebMapper indicates two applications running on the NARA website, but allots no size for them 




is anomalous. 



Table 6: Object Statistics 





Stat-USA 




NARA 




NIH 




Type 


Count 


Size 


Count 


Size 


Count 


Size 


Pages 


420 


515621 


261 


145219 


251 


509297 


Images 


17 


171392 


.84 


150407 


50 


38359 


Gateways 


7 


N/A 


2 


N/A 


3 


N/A 


Internet 


6 


N/A 


37 


N/A 


27 


N/A 


Java 


0 


0 


0 


0 


0 


0 


Applicatio 


66 


30263301 


2 


0 


0 


0 


ns 

Audio 


0 


0 


0 


0 


0 


0 


Video 


0 


0 


0 


0 


0 


0 


Text 


48 


821424 


5 


0 


0 


0 


Other 


348 


2952656 


0 


0 


1 


0 


media 

Total 


912 


34724394 


391 


295626 


332 


547656 



WebMapper generated a Status Summary (Table 7) which depicts hyperlink data. Taking Stat- 
USA as the first example, note that of the 876 onsite objects (pages, images, applications, etc .), 413 errors 
resulted. Such a high error rate may have a plausible explanation, but further investigation would be nec- 
essary to determine the reason. Several situations are possible, but the researchers lack error log files for 
Stat-USA and so cannot pinpoint the exact cause(s) of the high error count. The same report for NARA 
and NIK returned no errors. Stat-USA also returned 20 Not found 404 errors. Clearly these are defunct 
hyperlinks. A more detailed report provides URLs for each of these so that the website administrator could 
remedy these errors easily. 

Table 7: Status Summary 





Stat-USA 




NARA 




NIH 






Objects 


Links 


Objects 


Links 


Objects 


Links 


Onsite 


876 


1511 


299 


461 


220 


491 


OK 


443 


780 


48 


178 


59 


289 


Not Found 


20 


20 


0 


0 


0 


0 


(404) 

Other Errors 


413 


71 1 


0 


0 


0 


0 


Unverified 


0 


0 


251 


283 


161 


202 


Offsite 


36 


57 


92 


128 


112 


173 


OK 


28 


45 


4 


7 


8 


23 


Not Found 


0 


0 


0 


' 0 


1 


1 


(404) 

Other Errors 


2 


3 


0 


0 


0 


0 


Unverified 


6 


9 


88 


121 


103 


149 


Totals 


912 


1568 


391 


589 


332 


664 
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The bottom portion of Table 7 shows offsite links incorporated into the websites, meaning objects 
at different locations (domains) than its parent page. Given that the Web mappings described above all be- 
gan at the websites’ homepages, the offsite links truly are indicators of websites external to Stat-USA, 
NARA, or NIH. For example, the National Cancer Institute is one website nested within a complex of Na- 
tional Institutes of Health websites (with Public Health Administration above them, and the Department of 
Health above them all). Therefore most of the offsite hyperlinks are in fact crosslinks to other federal 
health agencies’ websites and online services. 

When WebMapper explores a website, it acts as a robot, or spider. It is the prerogative of the 
website being “x-rayed” by WebMapper to decline the privilege. One of WebMapper’s default settings is 
to honor “robot exclusion protocols.” Spider activity is rising rapidly across the Web, becoming one of the 
major forms of traffic and website visitations. When conducting log analysis website administrators should 
be cognizant of the likelihood that a portion of traffic through their sites are automated crawlers. Interest- 
ingly, despite the fact that Stat-USA is passworded due to its subscription nature, WebMapper had no 
problem mapping all pages within the website, passworded or not. The WebMapper application does not, 
however, retrieve the content of web pages, and so the security of Stat-USA was never breached. It is con- 
ceivable, by the same token, that some websites might hyperlink to sensitive data. Website administrators 
should be aware that present website administration technology such as WebMapper can provide a detailed 
structural view of any website that does not block Web spiders. It is quite possible that the inability of 
WebMapper to provide a structural overview of the large Census website was due to security features of 
that site. 

Web Server Log File Analysis 

By arrangement with Stat-USA, the study team used file transfer protocol to retrieve a 33 mega- 
byte log file from a Stat-USA server. Although the researchers expected a complete set of log files, they 
made do with a sample from the access_log. The access_log covered a period of Stat-USA website use 
from January 13, 1997 at 9:20 a.m. until January 21, 1997 at 10:07 a.m. (a little more then 8 days). To rep- 
resent analysis for a one-week unit, the team removed the last (eighth) day from the access_log. With the 
file on a local server, a team member ran customized PERL scripts against the access_log. The processing 



required about twenty minutes (the server used to analyze this data was a Pentium 150 MHz computer, 32 
MB of RAM, running on a shared Ethernet network, with a T3 backbone). The PERL scripts generated 
nearly 20 HTML pages, the data from which was imported into Microsoft Excel for analysis. The com- 
bined Stat-USA accessjog and HTML pages took up about 40 MB of storage space. The analysis shown 
below is only a fraction of what is possible to do with statistical tracking. Indications are made where fur- 
ther analysis and testing are required to approximate 100% accurate results. 

The Accessjog 

Access Jogs track information such as a computer’s Internet Protocol (IP) address (or domain name), 
which pages a user accessed, and at what times. It is possible to tell how long a user visited a site and what 
path the user took through a site. One limitation on log analysis emerges when a uses hits the “Back” but- 
ton on the browser to retrace steps. Accessjogs do not capture this action very effectively, and thus the 
trail through a website is formed by clicking on hyperlinks. Accessjogs are good at revealing: 

• Number of hits vs. number of accesses 

• Number of United States hits vs. outside countries 

• Path analysis of specific users. 

The researchers first compared the number of hits to the number of accesses. After running the 
analysis and studying the results, 
it was clear that a majority of the 
Stat-USA's accesses are database 
queries. However, the PERL 
scripts as written were in too 
simplistic a format and did not 
record database queries as ac- 
cesses. Rather than rewrite all 
the PERL script, a team member wrote a short shell script capable of analyzing the accessjog to provide a 
more accurate count of the number of accesses. 



Some basic definitions: 

Hit - A hit is any file from a website which a user down- 
loads. A hit can be a text document, image, movie, or a sound 
file. Say a user downloads a webpage that has six images on it, 
then that user hit the Web site seven times (6 images +1 text 
page). 

Access - An access, sometimes called a “page hit”, is an 
entire page downloaded by a user regardless of the number of 
images, sounds, or movies. Even if a user downloads a 
webpage that has six images on it, that user accessed only one 
page of the website. 
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Analysis of Stat-USA Access Log 



Comparative path analysis of specific users considered the following variables: 

• Number of hits vs. number of accesses 

• Number of United States hits vs. outside countries 

• Path analysis of specific users 

Several analytical uses can be made of the path (thread) a user navigates through a website, such as clock 
analysis, entry page, exit page, and more as described above. The table below represents a small sample of 
actual IP addresses or domain names that "hit” the Stat-USA server over the week of January 13 - 19, 1997. 
The table columns display the page at which users entered the Stat-USA site, what pages they saw while on 
the site, and finally the page from which they exited the site. At the bottom of the analysis is the time that 
each user spent on the Stat-USA site. Following that is a list of a few randomly picked users selected for 
path analysis. 

Figure 4: Webpage “Hits” versus Webserver Accesses 



Hits vs. Accesses 




Situated Proxies Assessment 

The ten situated assessments took place three sets of sessions, held at the users’ convenience. 
Three assessors looked at Stat-USA, three at the NARA site, and two apiece for NIH and Census. The 
sessions all went smoothly, and varied from about 40 minutes to an hour, depending on the user’s interest. 
On occasion, a researcher would intervene for one or another reason. These reasons ranged from asking 
the assessor to speak more loudly, or more specifically, to helping to attribute a slow download to either 
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the website or a Syracuse University server. In one instance, an assessor began criticizing the website’s 
choice of fonts. A study team member pointed out that font type and size could be altered in the Netscape 
browser preferences, and therefore might not be a issue generated by the website. Several times the situa- 
tion arose that the assessor might spend a large amount of time evaluating the websites’ search engine per- 
formance. While a perfectly valid thing to assess, website search engine effectiveness represented a minor 
portion of the evaluative criteria. If an assessor seemed to be spending a large amount of time analyzing 
searches, the researchers reminded them about the criteria categories pasted to the top of the monitor. Oth- 
erwise, the researchers said as little as they could and stood by to observe the searches and take notes. 
Some of the observations the researchers made of the assessors are: 

• All but two of the ten assessors rated their navigation and searching skills at four on a scale of 
five. The exceptions included a “three” and a “five.” However, during the assessment sessions 
the proxies clearly displayed more variance than indicated by their self ratings. The question 
about skill rating for the sample therefore failed to discern satisfactorily between users. The issue 
of skill itself however, is key. The more sophisticated a user was at conventional online searching 
(Boolean queries and judging relevance of the recall) the more adept they seemed at moving be- 
yond assessment of the search engine, and at distinguishing between browser preferences 
(settings) and website features. The study did not set out to delineate assessments according to 
user experience levels, but the data suggest that such a dimension may be important in analyzing 
user feedback, or more specifically, user assessments. The authors do not suggest that one cate- 
gory of user experience is preferable to another. Rather, that experience may play a role in what 
aspect(s) of assessment various users concentrate their primary efforts. 

• Most users were unable to distinguish between the features of the browser (Netscape) they were 
using and properties of the website, or of the possible mediation of the local (Syracuse University) 
server. This could result in unfair assignment of blame to the website. As in so many situations 
involving the Web, it is quite possible that the novelty and mercurial state of Web technology, and 
especially browsers at the present time, may defy a common understanding between users. In 
other words, the users’ general lack of recognition between browser and server characteristics may 
well be an historical effect. If a common browser interface emerges it could develop that famili- 
arity with the technology is less important to assessments. At the present time, technology is still 
very much in the foreground. It can be difficult to separate technology effects from information 
presentation. 

• Users’ searches (situated information needs) were often simplistic, or at best, highly general. Be- 
cause of this, any retrievals tended to be seen as a successful search. In one instance, a proxy 
found a satisfactory answer to his question within a minute of beginning his session. This is an 
indicator of success by almost any measure, and reflects credit on the website searched (Stat- 
USA). By the same token, it highlights a significant finding of the situated assessments: Presen- 
tation of information is most noticed when it interferes with a search. Most proxies focused on 
getting past the presentation of information to the information itself. This should not be surpris- 
ing, except it could suggest that cosmetic elements of a website are irrelevant unless they directly 
contribute to the access of information. Suffice it to say that this is contrary to the majority of 
commercial website marketing approaches. 






• Users seemed quick to accept the trustworthiness of information they retrieved from federal 

websites. Whether a function of low expectations or high precision of items retrieved, the proxies 
seldom questioned the reliability of their results. A follow-up (not done) might be illuminating. 
Although the proxies brought questions to address with them to their assessment sessions, in only 
a few instances did assessors bookmark or otherwise preserve information about their searches 
(even when deemed successful). This suggests to the researchers that most of the proxies did not 
intend to use their search results after the assessments (an assumption that may be unfounded — 
several proxies did indicate that they found the website interesting enough to return to later). 
Further comparisons of situated proxies with more naturalistic user samples could help clarify 
whether the situated proxies focused on answering their questions as a component of performing 
an assessment as opposed to assessing the website in the context of successfully addressing an in- 
formation need. 

The situated proxies assessed their respective federal website on a five-point Likert scale with “1” 
representing a low negative reaction, and “5” high positive (See Appendix 2G-H for criteria). Table 8 pre- 
sents results of the situated assessments. Ratings varied both by individual and by site. Usually, there was 
fairly high agreement between assessors looking at the same website. In a few cases, however, this was not 
the case. The mean overall ratings of the four websites were: NARA - 2.5, Census - 3.7, NIH - 3.6 and 
Stat-USA - 3.1. The ratings within each assessment category — Quality, Usefulness, Presentation, Naviga- 
tion and General — are consistent with the overall site ratings. Agreement between assessors was lowest in 
the Quality and Presentation variables, which recommends more attention be given to these categories. We 
hasten to note that the study was not designed to produce an authoritative coefficient of website, but rather 
a range of criteria that can provide the federal Web administrator guidelines. Many interpretations can be 
given for the figures listed above, and particularly since the websites are quite different from one another 
taking the values as fixed can be misleading. Having said all that, they can neither be summarily dismissed 
as irrelevant to user satisfaction. 

Comments made by situated users for each website include the following, first for NARA: 

• Major problem: Search engine not well marked. It is called NARA Information Locator 
(NAIL), and even then not clearly identified as a search engine. It was very difficult to locate. 

• The NAIL Search engine seems to retrieve a small amount of information considering what is 
available in paper format. This does not foster trust in the website. 

• Site severed connection on one user, although why. was unclear. 

• The information is we 11- formatted. 

• Downloads are fairly quick. 
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Summary: The NARA website has good information that is attractively presented, but the search engine is 
an Achilles’ Heel. 

The Census website received the most favorable review by the users. Because the researchers 
could not survey the Census website with WebMapper, the structure of the site is relatively unknown. It is 
a very large and complex site. The situated proxies said about the Census site: 

• Site holds good information. 

• It stimulates browsing. 

• Its content is attractively presented. 

• There are some formatting inconsistencies. 

• Some links suddenly take user offsite. 

• The User Manual is poor. 

Summary: Census was the best-received of the four sites. Users managed to get information from searches 
and menu options fairly efficiently. There were some problems with consistency of appearance and cur- 
rency of links. 

The National Institutes of Health website also impressed users, for the most part. Specifically, the 
assessors said: 

• Easy searchability. 

• It is well-laid out. 

• The menus are useful. 

• The search engine provided no explanation of its retrieval ranking system. 

• The graphics did not assist much in navigation. 

• In at least one case, a search returned as discrete hits three different parts of a single document. 

This was not clear until looking at the text of each hit (and there were only six total). 
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Table 9: Situated Assessment Data Summary 
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Stat-USA suffered from three primary problems: 

• Content is not novice friendly. 

• Inconsistent format (color changes, screen space wasted, columns not marked or flush, and more) 

• The Web aspect amounts to a thin shell. Data below the first layer appears to have been dumped 
in without links or careful formatting. 

Some frequent themes, both positive and negative, emerged from the situated assessments. Listed 
below are findings common to all or most website assessments and related to the Ease of Use and Content 
categories. 

Positive Findings: 

• Information on websites was substantial, useful, and perceived to be of fairly high quality. 

• Presentation of information on websites met an aesthetically pleasing standard. 

• Information returned from search queries generally proved satisfactory to information needs. 

• Layout of information, placement of banners and use of hyperlinks were generally consistent 
throughout each individual website. 

Negative Findings: 

• in some instances, proxies nau uirncuuy locating a searcn engine Decause ui poor laoeung or 
placement. 

• Help sections could be cumbersome. Too often they lacked clear and concise organization. 

• When hyperlinks took users to external websites, a problem often was lack of navigational tools on 
imported pages. 

• In several cases, the user’s ability to give feedback was either non-existent or limited. 

The ability to substantiate such important and informative information about a website communicates the 
effectiveness and usefulness of the situated assessments technique as a qualitatively valuable tool for fed- 
eral agencies in developing and improving their websites. 

Focus Group 

One week after the conclusion of the situated assessments all but one of the proxies met with four 
members of the study team to discuss impressions of the assessments, the methods involved, and sugges- 
tions for further research. The focus group lasted one hour and twenty minutes. Responses to probes 
submitted by the researchers to the proxies follow. 
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What constitutes good navigability? Members of the focus group felt that this was difficult to de- 
scribe because the parameters are not clearly understood. They said that one could always learn more 
about how to navigate the Web and the more time one spent at a site the better one becomes at navigating 
through it. One proxy maintained that good navigability was the efficiency at locating what one went to 
the site to find. Another suggested that a user of the Web has prior expectations of what they expect to find 
which can affect navigability. A couple of participants felt that their Web experience is above average and 
that the average user would have had more difficulty navigating the sites they looked at. 

How legitimate is the information on federal websites? Most participants felt that they did not 
have enough time to develop an opinion about information on the site they evaluated. Furthermore, it is 
interesting to note that participants generally felt, because they were at a government site the question of 
legitimate information was moot. In the case of STAT-USA however, proxies expressed frustration with 
the site’s lack of clear citations about sources of the statistical data. Even though most participants felt that 
it was difficult to judge the information because it being a government site, more than half (six people) 
stated that if this exercise was in a real world context they would have left the site to fulfill their informa- 
tion need elsewhere. 

Searching skills outside the Web environment. Many focus group members believed that their 
searching skills outside the Web were adequate, but were unsure of how the search engines on a site oper- 
ated (Boolean, Natural Language, etc.) which effected their results in locating information. Many partici- 
pants also complained that existing search engine help screens were unsuccessful in describing the search 
engine operation, and in the case of STAT-USA did not even deal specifically with that site. Limited time 
was also an element which participants said affected their searching results and overall evaluation of the 
site. Being able to simply locate the search engine was a problem for the National Archives and Records 
Administration site. Both individuals who evaluated this site commented on the difficulty they had in lo- 
cating the search engine because of the peculiar way it was labeled (“NAIL”). 

Are the methods used helpful to federal agencies? All focus group members agreed that the as- 
sessment methods used could help agencies improve their websites. One participant ventured that many 
agencies have been focusing only on creating a website without thinking about how the information on the 
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site might be used. Another participant stated that many agencies lack incentive to improve their sites, but 
that such incentives might be achieved through the situated assessment methods. The situated user ap- 
proach can further benefit an agency by offering critical feedback about their site's overall presentation and 
organization. By listening to users an agency may be able to gain insight about other potential users (and 
uses) of their site which could in turn, help them in developing and maintaining the site. Only two of the 
nine participants said that the site they visited fully matched their expectations. This position seems to 
contrast with the majority of proxies’ stated satisfaction with their search results and confidence in the in- 
formation retrieved. 

Are criteria transferable to other websites? Participants unanimously concurred that the criteria 
could be applied to practically any website. Every website faces issues of presentation, navigation, quality, 
and usefulness. Participants further noted that for every site there is a target audience and knowing your 
target audience can improve the site development as well as support the evaluation of a website. 

Was the sample appropriate? No one expressed any reservations about the use of situated prox- 
ies, or about the mixture of experience levels in the assessor sample. On the contrary, the group agreed 
that the utilization of Web-experienced individuals provided a vaiia test group. A few focus group mem- 
bers thought that it should be possible to learn even more from the situated assessments by contrasting ex- 
periences of novice users with experienced Webmasters. 

Findings 

Findings from both the situated assessments and subsequent focus group offer insight into the use- 
fulness of these techniques in evaluating federal websites. Some specific findings include: 

• Proxies more familiar with Web browsers better identify the most likely sources of data transmis- 
sion problems, and are not reluctant to attribute delays to the website they are evaluating. 

• It is important to carefully distinguish between proxies’ hypertext navigation skills and more tradi- 
tional (e.g., Boolean) searching skills. Hypertext searching is typically point-and-click navigation, 
which may differ markedly from using a search engine. The present study did not set out to evalu- 
ate the selected websites’ search engines, yet often those were often aspects of the sites on which 
many users focused heavily. 

• All proxies agreed that time allocated for the search session (45 minutes) was insufficient to form a 
thorough opinion about the quality / reliability of information found on the website. 

• Proxies’ Web experience and familiarity with the type of content available on the website aided 
their ability to evaluate results of their searches. 
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• All proxies agreed that the current assessment methods is a useful technique for federal agencies to 
use to evaluate their websites. 

• Proxies agreed that not only is this method useful to federal agencies, but it could probably be ex- 
tended to other types of websites. 

These findings underscore the usefulness of the situated assessment as a valuable tool in assessing 
websites, and offer substantial recommendations for the improvement of the technique overall. 

User-Based Guidelines 

Web users risk giving too much faith to the powerful new medium. They receive information 
from a search, so they may feel relatively satisfied. Often they seem not to take the additional step of 
evaluating the quality of the information obtained. Perhaps a special case of separating technology from 
information presentation is separating information presentation (and presence) from information quality. Is 
technological transparency resulting in information opacity? One of the incentives for conducting this 
study is the researchers’ awareness that a tremendous amount of “information” is being loaded very rapidly 
onto the Web. A less sanguine view might say we are participant-observers of a “data dump” of epic pro- 
portions. Information availability can be subsumed by information obliteration. The rise of websites as 
public information dissemination vehicles reinforces a need to develop quality criteria for assessing federal 
information and services available from federal websites. The best way to ensure that the utilitarian agenda 
of universal access is met, is to bring users directly into the design and maintenance processes of systems 
development. 

Powerful organizational and technological processes are converging to produce information tech- 
nologies that will play crucial roles in providing, or limiting access to public information. By providing 
ongoing user feedback to those responsible for the development of government information systems, more 
robust systems may result. User assessments can warn systems developers of problems before they be- 
come inextricably embedded. Similarly, user assessments can help to ensure that information systems are 
driven by user needs rather than user needs being determined by technological and organizational impera- 
tives. 
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Technical Assessments 



One of the principle goals of this project was to develop technical and policy assessment instru- 
ments that could easily be employed by federal agencies to evaluate the quality of their websites. Unlike 
the “situated” assessment process, which attempts to ascertain user perceptions regarding content quality, 
organization, usefulness, and overall ease-of-use, the technical and policy assessments were intended to as- 
certain in a more objective manner, the site’s compliance with accepted technical criteria associated with 
high-quality websites and policy criteria relevant to federal websites. 

One member of the study team pretested the Technical and Policy Assessment criteria and proto- 
cols (See Appendix 3) . The reviewer accessed approximately 25 pages at each of the four sites, conducted 
at least one search, and visited several help pages. The reviewer supplemented the structured responses to 
the assessment criteria with marginalia. The researcher discovered difficulties with both instruments. The 
problems were most evident with the Technical Assessment which the researchers had expected to be fairly 
straightforward, but similar difficulties arose during the Policy Assessment. In order to verify these pre- 
liminary results, two other members of the study team repeated the assessments with similar outcomes. 

After meeting to revise the Technical and Policy Assessment procedures, two study team mem- 
bers reassessed each website with the new instruments (Appendix 3). In both the pretest and the subse- 
quent evaluations, reviewers assessed each site for approximately one hour, over a minimum of five pages 
per site. The pages selected for review came from major links available on the websites’ homepages. Al- 
though adjustments to the procedure yielded somewhat improved results, at the conclusion of this study 
there are still issues to be addressed in developing effective assessment instruments that attempt to assess 
complex websites by sampling pages rather than comprehensively reviewing the entire site. Even a com- 
prehensive assessment would pose problems of decontextualization. The following sections provide details 
of the difficulties encountered in the Technical and Policy Assessments. 

Findings 

Appendices 3A-D display the results of the technical assessments. As will be discussed in the Is- 
sues and Recommendations section, the reviewers were unable, in many instances, to complete the assess- 
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ment instrument. The completed technical assessment forms (Appendices 3A-D) include many question 
marks reflecting problems in drawing yes/no conclusions about a site’s features. Due to a large degree of 
complexity within the websites considered by this study, to make fair and accurate assessments many cri- 
teria have to be answered differently according to the particular page. One finding of this study is that 
consistency within a website is a very difficult variable to measure. Websites typically include hyperlinks 
to external sites which take the user into radically different environments than the host site, and, within any 
single site the informational context may vary significantly requiring different measurement yardsticks, or 
at the very least, individual treatment. A key difficulty with the technical assessment approach used in this 
study was the attempt to assess websites at a holistic level, when they are frequently highly variegated ag- 
gregates. Having made this point however, the criteria themselves seemed to handle individual webpages 
well. The levels of analysis issue must be addressed in any future assessment effort. 

Despite these limitations, each of the examined sites met most of the identified criteria. Due the 
heterogeneity of websites, the researchers found assessment of the following areas to be especially prob- 
lematic: 
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• availability and consistency of navigation buttons (return to homepage) 

• functionality of navigation bars 

• availability of downloading options and information about file sizes 

• consistent distinctions between graphics and graphical lengths 

• anchors in lengthy documents or breaking them to ensure they did not exceed three screen lengths. 
The sites did not generally do a good job of ensuring that users could remain oriented within the sites and 
providing fully functional navigation aides. 

Recognizing the limitation of applying many of the technical criteria to the website as a whole, the 
other two reviewers both selected five pages at each site and assessed those pages independently. Their re- 
sults validated the initial assessor’s. To remedy the methodological problems with this aspect of the as- 
sessment either a comprehensive assessment of entire websites is necessary (which may be impractical in 
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many cases), or a very sophisticated sampling technique with assessment criteria typologized according to 
the characteristics and intended uses of the retrieved webpages is required. 

Policy Assessment 

Appendix 3C summarizes assessments by three reviewers. All of the federal websites assessed 
largely complied with existing information policies. The content and services included on these sites were 
consistent with agency missions and functions. With possibly one exception, the sites examined did not 
appear to advocate political positions. The possible exception — the Tiger Mapping Service at the Census 
Bureau site — did not explicitly advocate a political position but warns that the service may be discontinued 
due to funding limitations. This might be interpreted as lobbying, but could also be defended as fair 
warning to users. The external links to commercial sites included on these sites appeared consistent with 
the site’s intended use. 

With the exception of the NIH site, these sites provided visitors to their sites a means of contacting 
Web administrators, and in some instances the actual content providers. For the most part, site visitors are 
able to assess the currency of the posted information. None of the websites appeared to be engaged in data 
collection. 

The criterion on the policy instrument: “Are the location aids GILS compliant?” requires better 
definition. Reviewers found it hard to interpret as stated. Members of the research team participated in a 
recent evaluation of the Government Information Locator Service implementation and are positioned to 
revise the GILS criterion for website assessment purposes. 7 Depending on the outcomes of the evaluation, 
a core set of criteria may be developed which target specific aspects of federal information dissemination 
through websites. Central to formulating a set of GILS core criteria to be applied to websites are two re- 
lated questions. First, what constitutes an official record in a website environment? This question is being 
researched by McClure, et. al , under a contract from the U.S. National Historical Publications and Records 
Commission (McClure and Sprehe, 1996). Mirroring that issue is the problem of characterizing Web envi- 



7 An Evaluation of the Federal Government 's Implementation of the Government Information Locator Service (GILS). 
(Sept. 26, 1996). W.E. Moen and C.R. McClure. Work Plan submitted to the General Services Administration, Office of 
Information Technology Integration, Washington, D.C. Contract no. GS00K96APC2368. Final Report pending, Spring 
1997. 
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ronments under any accurate and general definition. Websites are both diverse and dynamic. And that 
within a single site. When surveying the conceptual geography of federal websites, a shifting, composite 
picture emerges. GILS has potential utility for identifying at least a minimum set of features every official 
information dissemination vehicle should have. Perhaps the convergence of the two questions just de- 
scribed will occur if and when federal agencies begin trying to characterize their websites according to the 
GILS profile and core elements. This is an area that may hold prospects for further research. 

Findings 

The reviewers found the sites generally in compliance with federal policies and with technical 
guidelines associated with high quality websites. The reviewers did note some variation in the level of 
technical sophistication and policy compliance from one website to another, and the potential for im- 
provements exists at them all. The reviewers report some problems with the design of the assessment in- 
struments and evaluation procedures. The Issues and Recommendations section discusses how the 
Technical and Policy assessment processes can be improved. 

Site Visit 

Members of the study team visited Stat-USA in its offices at the Department of Commerce, in 
Washington, D.C., on February 21, 1997. The visit lasted most of a day and consisted of several inter- 
views followed by a focus group, attended by four researchers and five Stat-USA staff. Topics ranged 
from organizational alignments to technology requirements, and from user/client issues to technical exper- 
tise required of staff. See Appendix 4A for probes presented to Stat-USA during the site visit. During the 
midpoint of the focus group the researchers showed an 1 1 -minute videotape of situated users assessing the 
Stat-USA website. 

Stat-USA Internet staff described the Web-based operation as “the federal government’s premier 
electronic publisher of trade and economic information.” 8 Stat-USA is unusual in that it provides sub- 
scription access to its databases which in turn, funds staff salaries and pays for overhead and equipment 
costs. The relative autonomy from its parent organization (the Department of Commerce) allows this small 



From a promotional brochure produced by Stat-USA. No title. #12/96- 10k. 
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(fewer than 30 employees) operation’s website to be highly flexible, and according to its administrators, 
highly responsive to its clientele. Stat-USA operates as a small business, including a philosophy of provid- 
ing value added products to its customers. Stat-USA claims authority for following a business model from 
Vice-President A1 Gore’s National Performance Review (1994). The website’s chief administrator calls 
the site “an experiment” which is successful only so long as it manages to “break even” on recovering 
costs. Although the Stat-USA site is in many respects rich and challenging to assess, it is nevertheless 
atypical of other federal websites. For this reason, some data collected during the course of the site visit 
are not generalizable to other federal sites. The researchers have elected to limit analysis and to report 
findings to areas that may be compared to the other selected federal websites. 

Findings 

Below are observations from an interview which pertained to Stat-USA user charges and websites 
finances. The participants were the two most senior members of Stat-USA, and three members of the 
Syracuse University research team. 

• Website users are largely high-end, or corporate and other government agencies. 

• Promotional activities are ongoing to expand user base. 

• The website aims to “break even.” During its first year of operation it lost over $200,000, but it 
recovered costs its second year. 

• If subscriptions exceed costs in any given year, the funds will be used to upgrade facilities, 
equipment, staff and services. 

• The Stat-USA website operates as a “small business” along lines suggested by the 1994 National 
Performance Review. The administrators observed that the Stat-USA website is an organiza- 
tional experiment that may become more common over the next few years. 

• At the time of the study, Stat-USA had not systematically assessed its website from a user per- 
spective. 

• Stat-USA website administrators did not regard a lack of policy guidance from OMB, NARA or 
other information management agencies to be a problem. On the contrary, they maintained that a 
looser policy environment allows a flexibility well-matched to the dynamism of the current Web, 
and best suited to meeting client needs. 

• While Stat-USA staff said they found the videotaped remarks made by situated assessors to be 
enlightening, but voiced skepticism whether the substance of the proxies’ concerns would accu- 
rately reflect those of actual users (subscribers) of the website. 

• Stat-USA has already revamped much of its website into a LotusNotes format that appears to im- 
prove on several areas criticized by situated users. 
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Stat-USA’s unusual (at least in the federal government) financial and organizational structures 
allow it to succeed at delivering information products. Whether the model being pioneered by Stat-USA is 
transferable to other websites is an open question. More study is required to examine costing issues of fed- 
eral websites providing fee-based services. 

During the site visit it became clear that one of the intended data collection activities - online 
popups — would be difficult to arrange. The website administrators felt that such an event interrupting the 
site users’ sessions would be too intrusive, and distract users from their primary purposes. The Web ad- 
ministrators maintained that most subscribers to the site were familiar with the website and knew how ef- 
ficiently to obtain what they wanted. Popups would defeat the assessment purpose by drawing negative 
commentary to their own presence, rather than to the website itself, but the negative effect would accrue to 
the website by association. 

The discussion of popups highlighted an intriguing and related situation with the Stat-USA 
website. The Stat-USA administrators believed, as already stated, that they knew who their users are, as 
indeed a subscription service would, and that informal means of feedback served the site well. When asked 
whether they relied on email to obtain feedback they said they did not. Email feedback would come from 
the email address (hyperlinked) which appears on the bottom of nearly every page of the website. The user 
could click on the link to bring up an email form that seemed addressed to the Webmaster. However, eve- 
ryone at Stat-USA agreed that based on their past experience with this communication channel, they re- 
ceived more noise than message, and so presently do not seriously consider email to be useful for feedback. 
Since casual browsing is not relevant to the website, given its subscriber base and business nature, it may 
well be that the informal network and the telephone are more powerful means than email for gaining audi- 
ence response to this site. 

Dissemination of Findings 

The principal investigators were invited to present their findings to a consortium of high-level 
government executives interested in the emerging prominence of the Web in federal information manage- 
ment and dissemination. As part of both the dissemination effort and verification of the study’s findings, a 
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member of the study team presented an overview of the evaluation techniques developed in the study and 
findings to federal officials at two meetings in Washington D.C. in April 1997. About 75 individuals, in 
total attended these two meetings. Most had responsibilities for web administration at their agency, al- 
though others included agency heads, systems administrators, and records managers. 

The presentations stressed the overall importance of federal websites evaluation, and networked 
information resources and services in general. Beyond a discussion of evaluation basics the presentations 
included specific explanations of: 

• the online situated user assessment 

• Web server log analysis techniques 

• the use of focus groups in the study. 

During breaks and informally, additional discussion occurred regarding webmapping and use of video 
taping in the situated online user assessment. 

Participants were very interested in these assessment techniques and asked probing, knowledge- 
able questions about the evaluation techniques and how best to use them in their particular settings. The 



various iog analysis techniques incurred a great deal of interest — especially in their potential for cross-log 



analysis techniques. Additional discussion of privacy issues and the possibility that agency log files could 
be accessed through the Freedom of Information Act process generated considerable interest. 

The federal attendees expressed keen interest in the researchers' various evaluation techniques and 
described their need for ’’turn key” approaches that are, as one individual described them "quick and 
clean.". They especially encouraged that all additional work in the area of evaluation be reported and were 
interested in obtaining final reports of both this project and similar work recently done on the Government 
Information Locator Service (GILS), 9 which also developed interesting applications for web assessments. 

Conference participants also noted that the development of evaluation tools such as those reported 
at these sessions could provide an effective means for them to meet the requirements of the Government 
Performance Results Act (GPRA) of 1993. The GPRA requires agencies to develop goals and performance 



9 An Evaluation of the Federal Government's Implementation of the Government Information Locator Service (GILS). 2 
vols. W.E. Moen & C. R. McClure. Denton, TX: University of North Texas, School of Library and Information Sci- 
ences, 1997. (Forthcoming in ERIC Clearinghouse). 
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indicators for key services and activities -- such as Web-based services. Indeed, participants generally 
agreed with study team findings pointing out the lack of policy guidelines for federal web management and 
electronic records management in the federal web environment. 

The responses from the agency individuals attending these sessions were extremely positive. 
Further, their comments and suggestions for future research verified the importance of developing evalua- 
tion techniques for web-based services. While they were pleased with the work done to date, their com- 
ments also informed the study team's perceptions of areas where additional research should be considered 
in the broad topic of evaluation of web-based services and resources (discussed in Chapter 5). Clearly, the 
work reported in this study was of key interest to these participants. 

In addition to the study techniques and findings presented at the two federal forums in April, the 
study team has taken other dissemination actions. To date these include a detailed treatment of the policy 
context of the assessment of federal websites published in the Spring 1997 issue of Government Informa- 
tion Quarterly (Eschenfelder, et. al.). A draft of preliminary findings was accepted for inclusion on the 
program (and in the Proceedings) of the American Society for Information Scientists Annual Conference, 
to be held in Washington, D.C., from November 01-05, 1997 (see Wyman, McClure, Beachboard and 
Eschenfelder, 1997). Additional dissemination activities will be performed as appropriate. One of these 
will be making available the Final Report through the ERIC Clearinghouse at Syracuse University. 
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Chapter 5. Issues and Recommendations 

This study developed and evaluated system-, user-, design- (policy and technical), and developer- 
based criteria for evaluating federal websites through a variety of assessment methods. The previous sec- 
tions described the methods and findings resulting from application of a multimethod approach. The as- 
sessments identified several issues that should prove useful for federal officials charged with developing or 
evaluating Web-based information dissemination services. 

Objective Questions 

In contrast to the user-based situated assessments, the study team intended the technical assess- 
ments to be as objective as possible. The majority of the assessment questions were framed simple yes or 
no responses. For example, “Is a text only version of this site available?”, or “Does the site consistently 
differentiate between graphics and graphical links?” Despite the researchers expectations, some questions 
proved to be relatively subjective; for example, “Do hotlink headings and descriptions match the informa- 
tion found through the hotlinks?” or “Are banners consistently designed?” Facts cannot speak for them- 
selves. All criteria need to be interpreted so it is not possible to escape exercising some degree of 
subjectivity. The technical criteria, without detailed operationalized definitions, can produce incommen- 
surable interpretations between raters resulting in reliability problems. The study team was unable to 
complete the multiple assessments required measure inter-coder reliability. The difficulty the assessors had 
in deciding how to answer some of these questions exposed the complexity of assessing heterogeneous 
websites based on small page samples. Although the policy assessment is fairly prone to the same opera- 
tional problems, the study team never expected a simple and objective standard for that assessment. 

Another issue affected the objectivity of the technical and policy evaluations. As discussed briefly 
above, some of the criteria assess specific elements such as webpages or hyperlinks. The websites assessed 
contained numerous pages and links, and so criteria seldom could be answered with a simple yes or no. 
Instead, the evaluator had to make judgments regarding the extent to which the site complied with given 
criteria, and then force a binary decision, yes or no, when most sites were neither fully compliant nor fully 
non-compliant. 
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Recommendation: The evaluation procedure should specify the extent to which the site will be 
evaluated ( e.g ., the number of pages, documents, links, etc.) and explicitly include criteria for 
evaluating the extent of compliance ( e.g ., 90% pages in compliance would rate outstanding, 80% ex- 
cellent, 70% good, etc.). 

System Dependencies 

A second issue germane to all the evaluations pertains to criteria which may produce varying re- 
sponses depending on the hardware and software used. One criterion specifies that webpages should not 
exceed three screen lengths. Conformance to this criterion can depend on the size and resolution of the 
monitor, as well as the configuration of the Web browser. Additionally, while some websites did depict 
graphical hyperlinks by highlighting them in blue, in other instances the highlighting did not display well. 
This problem may well have been a function of the resolution and color settings of the client system. The 
federal consortium specifically advises in its - guidelines that federal websites, to the extent possible, should 
work with all Web browsers and on a variety of hardware. Given the range of hardware and software 
configurations available, this advice represents an ideal. 

Recommendation: website evaluation should include the use of a variety of system configurations, 
browsers, and communications links. It is neither desirable nor practical to test all possible configu- 
rations, yet it should be possible to test both generic low and high-end systems using the leading 
browsers. 

Assessing a Moving Target 

The technical and policy assessments revealed the importance that timing can play in assessing a 
website, particularly in conducting an external assessment. The researchers assessed the Stat-USA website, 
and then several days later while downloading several pages to review discovered that significant changes 
(at least in terms of the evaluation) had since occurred. Though the look and feel of the site remained 
largely the same, the Web administrator had apparently gone through the site and redesigned many of its 
pages to be consistent with the guideline that ‘Webpages should not be longer than three screens,’ and, in- 
corporated some of the new frame-based design features that can improve the appearance and usability of 
the site. Clearly, many websites are undergoing near continuous change. 

Conceptualizing Websites 

Among the most difficult issues to emerge during the technical and policy assessments concerned 
the identification of the proper object or objects to which criteria referred. The intent was to provide a 
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means for assessing federal websites, but in doing the assessments it became clear that for some websites a 
singular assessment of the entire site could not produce fair and comprehensive outcomes. Some sort of 
generalized taxonomy of federal website form and function is necessary for the approach attempted in this 
study to work most effectively. Short term alternatives to the classification approach could be evaluating 
only according to elements a website should and/or should not have comprehensively for the entire website 
(all pages). Such an approach deals only with matters of kind, and does not recognize matters of degree. 
Otherwise, a more articulated sampling procedure might be devised to collect a representative collection of 
webpages for complete assessment. 

Federal websites can be massive. Realizing that it would not be practical to assess every web 
page, the study team elected to identify five pages to assess and generalize the results to the website as a 
whole. In retrospect, for some sites this approach may not have provided an adequate characterization of 
the websites as research constructs. Is a website an organizational construct, for example, and are the 
component webpages maintained by a single Web administrator? Is it a system (hardware and or software) 
construct operating on a single, or set of, closely coupled Web servers? Is a website more properly con- 
ceived as a service construct where a coherent collection of information and services constitute the object 
of interest - regardless of how the collection is mounted on the Web or administered? The nature of the 
Web is such that any of these alternatives could prove useful. From federal policy and assessment per- 
spectives there needs to be careful consideration of how the website or service is to be conceptualized. 

The issues described above affect technical assessments of federal websites because of the range 
of content and information services that can be provided by a single federal agency. Design guidelines 
typically call for establishing a consistent “look and feel” for websites. The Census Bureau website, for 
instance, has done a good job of complying with this provision. However, webpages related to the Tiger 
Mapping Service (example pages are included in Appendix 1A) established a distinct look and feel as ex- 
emplified by the use of a tiger logo and the inclusion of a return to homepage link at the bottom of these 
pages returning users to the Tiger Mapping Service homepage rather than the Census Bureau homepage. 
The operation of a service within a service (or perhaps more accurately stated, a website within a website) 
may be perfectly legitimate, and may in fact be one of the particular strengths of websites. In the context 
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of the assessment, however, the procedures used did not adequately account for this organizational rela- 
tionship. 

Recommendation: Consistent with the recommendation provided above, prior to conducting an as- 
sessment, evaluators should coordinate with the website administrators to determine if the website 
hosts (or points to) distinct services that merit separate evaluations. To a large degree, software such 
as WebMapper used in this study can provide a great deal of detail in advance of technical or user as- 
sessments to evaluators. 



Conceptualizing Webpages 

Because of the hyperlink capabilities of the Web, webpages can serve as indexes — pointing di- 
rectly to content or to other indexes — or to combinations of content and indexing. Combining content and 
indexing on web pages represents high potential for the Web, but it comes at a significant cost. Web 
authoring tools are making it increasingly easy to convert documents originally prepared for print publica- 
tion into HTML formats, and then create index pages that point to the electronic version. Unfortunately, 
the electronic documents seldom include the standard banners or navigation bars, or conform to the three- 
screen and anchoring criteria recommended for Web publication. 

Due to their desire to make a wide variety of information accessible via the Web, many federal 
agencies can be criticized for failing to conform their websites to the aforementioned criteria. Yet, if re- 
source limitations force a choice between having a greater number of minimally formatted HTML infor- 
mation available, as opposed to a more limited number of fully compliant documents specifically authored 
for the Web, it might be preferable to err in favor of increasing the amount of content. Of course, federal 
Web administrators will need to ascertain the needs and desires of their target audiences in determining the 
appropriate balance between these two alternatives. 

Recommendation: It would seem that different criteria might be required for evaluating web pages 
serving the distinct functions described above. Additionally, given the inevitability of restricted re- 
sources, agencies should confer with key stakeholders to determine whether they should maximize 
the amount of content available via the Web (implying minimal Web-specific authoring) versus limit- 
ing the amount of content made available, while still investing resources to produce quality informa- 
tion designed to take advantage of Web technology . 10 



10 This discussion of potential webpage categories is certainly not exhaustive. For example, webpages that solicit infor- 
mation, provide searching aides or user assistance might all merit the development of specific assessment criteria. 



Selection of Users / Assessors 



The selection of assessors can significantly influence the results of the evaluation and the quality 

of information provided. While federal websites have and implicit obligation to serve the information 

needs of all U.S. citizens, it is not possible to tailor websites to meet all the needs and preferences of such 

a disparate set of users — disparate not only in terms of education and cognitive need, but also in terms of 

how and for what purposes the website is accessed (and if they have the capability to access it). While 

some users may prefer highly formatted content optimized for online browsing, others may want minimally 

formatted text suitable for downloading and offline applications. Federal Web administrators should also 

consider the needs of the disabled users when designing websites. 

Recommendation: Federal website administrators should identify target audiences and select user / 
assessors that best reflect those audiences. It may be that multiple audiences are being served and a 
sample of users from each audience could be selected to perform website evaluations. With respect 
to use of qualitative techniques, selection of exemplar users with actual information needs is more 
important than attempting to select a statistically representative sample form the general population. 

Use of Structured Versus Unstructured Assessment Techniques 
The various instruments developed and evaluated in this study provide a relatively parsimonious 
means for assessing general satisfaction with federal websites. The instruments permit Web administrators 
to collect relevant data from a large number of users at fairly low cost. One should be cautious in reporting 
the results of such studies, though. While the instruments themselves displayed an acceptable level of ex- 
ternal validity, the difficulties inherent in drawing statistically representative samples form the population 
should not be underestimated. 

Conversely, most qualitative evaluation techniques do not attempt to obtain statistically represen- 
tative samples. The lack of statistical sampling does not imply that it is impossible to make valid infer- 
ences to a more general population, however. The real strength of using more qualitative methods lies in 
the type of information that can be produced. The “Think Aloud” protocol that was used provided more 
that an assessment of user satisfaction: It provided a means for observing the actual use or navigation of 
the website, identifying where and what types of problems the evaluator experienced, as well as which 
portions of the website worked well. 



Recommendation: Federal website administrators should identify target audiences and select assessors 
either from, or who fairly represent those audiences. A single website may serve multiple constituencies, 
and when such is the case samples of users from various assessors could perform website evaluations. 

Resource Limitations and the Need to Assess Federal Websites 
In an environment where resources are constrained, meaning nearly all, one can easily rationalize 
that limited resources are better spent posting content available on the Web rather than evaluating how the 
website gets used, particularly in light of the electronic Freedom of Information Act provisions mandating 
federal agencies to provide access to electronic versions of specific types of information. However, there is 
some merit to the saying that: “If it’s worth doing, it is worth doing well.” Without some level of evalua- 
tion, it simply is not possible to tell whether an acceptable, much less optimal, level of service is being 
provided. 

Although the assessment methods and instruments described in this report require refmement, 
they provide strong bases for conducting internal evaluations. If an agency’s ability to recruit external re- 
viewers is for some reason limited, it may be possible to recruit a limited number of in-house personnel 
(preferably not the website developers, administrators, or content providers, though each may have their 
voice) to perform a limited number of user-based qualitative assessments. 

Recommendation: Federal website administrators should not permit a lack of resources needed to 
conduct a comprehensive website assessment prevent them form attempting to perform any evalua- 
tion at all. Even a small assessment conducted rigorously and honestly can provide information that 
can be used to significantly increase the usefulness and quality of the website. 

Recommendations for Future Research 

The study was exploratory in nature and accordingly identified questions and issues for conduct- 
ing further research. Below are key issues to be addressed: 

• Testing internal validity (convergent / discriminant validity) and external validity for the struc- 
tured instrumentation continues to be an important issue. With regard to external validity further 
research is required to determine to what degree conformance with technical criteria is associated 
with user satisfaction. 

• Additional clarification is needed on the concepts of website and webpage. How should appro- 
priate units of analysis be determined? Should different criteria, or criteria weightings, be applied 
for web pages serving different functions? 

• The various types of log analyses offer significant opportunities for research. Of great interest to 
this study team is exploration of cross log analysis techniques, i.e., using data from multiple logs 
in sophisticated statistical analysis programs. 
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• Understanding individual versus group preferences for websites is an interesting research prob- 
lem. Webmasters are often forced to design sites and pages for “the greatest good.” But in fact, 
this study found a wide range of idiosyncratic preferences for specific aspects of websites and 
pages. 

These research topics are identified as the most interesting ones to pursue from the perspective of the study 
team. Clearly, numerous others also appeared during the course of the study. 

During the site visit to Stat-USA, the researchers asked the administrators what they foresaw as 
top priorities for the website in the near future. Without hesitation they responded that “push technology” 
was emerging as an important feature on the horizon (McGarvey, 1997; Novak and Hoffman, 1996: 37). 11 
Stat-USA currently serves as a website repository of statistical data. The staff of Stat-USA envision being 
able to “market” the data to potential subscribers, as well as deliver packaged content online as an active 
service. Building push technology into their federal website is a way in which the Web is advancing new 
models into government information management and dissemination. Stat-USA’ s entrepreneurial experi- 
ment will move into a new phase if indeed the website begins niche marketing using new intelligent push 
technology. Not everyone, it should be noted, is as sanguine about the potential of push technology 
(Brenner, 1997: 11). 



As more Federal agencies invest in websites to disseminate information and provide services, 
questions concerning the selection and configuration of appropriate Web technologies, the appropriate 
types and levels of services to provide over such websites, the types of information made available, and a 
range of related questions will continue to be asked. Increasingly, policymakers, as well as users, are ask- 
ing questions — not about whether their agency should invest website development — but about ways to 
maximize the benefits and impacts of existing Web initiatives. 

The notion of the "networked environment" encompasses a range of electronic networked activi- 
ties and services. Minimally, the networked environment includes information services, products, hard- 
ware and software, telecommunications infrastructure, and resources that users and providers of the 



11 Push technology is a means of actively delivering information to a user (presumably a subscriber to an information de- 
livery service), instead of presenting the same information on a website and requiring that the user come to it. Synonyms 
include pointcasting, narrowcasting, niche marketing and direct marketing. 



Importance of User-Based Evaluation 
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network receive via the various electronic networks. Increasingly, Federal agencies are relying on Web- 
based technologies to provide a range of information and services. 

To a large extent, proponents for developing Web-based services have engaged in a “technology 
as a magic bullet” approach, claiming that websites improve a person’s quality of life, improve overall 
productivity, and enhance the competitiveness of an organization — among other things. But the reality is 
that hard evidence to support such assertions is extremely limited or anecdotal. In times of budget cuts and 
federal retrenchment (such as we are seeing today), proponents of Web-based services will find it increas- 
ingly difficult to justify expenses for deploying such technology, without also being able to demonstrate 
that Web-based services really have some impact in meeting specific individual or organizational objec- 
tives. 



In this Web-based environment, however, there is the potential to provide a huge array of infor- 
mation products and services. The concept of Web-based information services is an evolving one. As- 
sessment techniques, however, related to Web-based information services by specific audiences such as 
government officials, teachers, business persons, researchers, students, librarians, etc . are just in the devel- 
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• What are the scope and reach of Web-based information services? 

• How much use do websites and individual pages receive and why are they used? 

• What types of users access the website and to what types of services and activities do they con- 

nect? 

• What are the costs for providing various types of Web-based services? 

• How has access to and use of Web-based services affected organizations and individuals in terms 

of economic competitiveness, education, productivity, quality of life, and other traditional per- 
formance indicators? 

• What types of performance measures are appropriate to determine the impact of Web-based 
services on various stakeholder groups? 

• What types of evaluation tools can assist policymakers website managers, and researchers meas- 
ure those identified performance measures? 

To date, there has been little practical guidance offered to assess federal websites and Web-based services. 
As agencies implement new information technologies and expand Web-based services there will also be a 
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need for evaluation methods and measures to demonstrate that the resources invested in such efforts have 



had some positive impact on the users they serve. 

The work reported in this study suggests that developing assessment techniques for federal 
websites and Web-based services is a multi-dimensional process due to the complex technological nature 
of Webs and the Web environment as well as the complex behavior and information needs of users. 
Nonetheless, the report offers several methods to evaluate federal websites and Web-based services, and 
identifies and discusses key issues and findings that affect the successful evaluation of such services. 

The efforts discussed in this report offer a beginning framework for describing, defining, and op- 
erationalizing assessment tools for federal websites and Web-based services — with an emphasis on as- 
sessment from a user perspective. As noted by Brigman (1997, Supplement 3): 

The single most important issue for any website is taking the time to step back 
from your vision of yourself as a company or a department, and seeing the site 
as your visitors do. 

This perspective is an underlying theme throughout the study reported here. Within the context of this 
theme, the study offers assessment techniques to evaluate interactions and services within this environment 
so that researchers, users, decision makers, and policymakers do not have to guess at what seems to work 
well and why. Although the federal website and Web-based services environment is dynamic and con- 
stantly evolving, there is a need for a core set of measures and assessment techniques that evaluate Web- 
based services, activities, and technologies. Strategies suggested in this report offer a first step in meeting 
this objective. 
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APPENDICES 



1. SYSTEM-BASED ASSESSMENTS 



A. Website Mapping 

B. Web Server Log File Analysis 



l.A. WEBSITE MAPPING 
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NetCarta WebMapper Home Page 



http://caIlisto.syr.edu/wmux/ 




NetCarta WebMapper™ Home Page 




Navigate either from the icons above or the main menu shown below. 



Ifffil New Map from URL: Create new map from URL and generate reports 

l|9l New Map from File: Create new map from file and generate reports 

Ilf 1 Open Map: Open map and generate reports 

ill Compare: Compare two maps and generate reports 

HP| Verify: Verify links 

Bl Report: View saved reports 




: View help information 
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U.S. Census Bureau Home Page 



http://www.census.gov/ 



topics in text 







About the Bureau 
User Manual 
New on the Site 



Current U,S, Qirrent 





<1 Other Official Statistics 
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The study team was unable to produce a website map of the Census website (http://www.census.gov/). 
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National Archives and Records Administration 



http://www.nara.gov/ 




WELCOME! 



From the Archivist 



NARA's Organization 



Welcome to the National Archives and Records Administration (NARA). 
NARA is the government agency responsible for overseeing the management 
of the records of the federal government. NARA ensures, for the Citizen and 
the Public Servant, for the President and the Congress and the Courts, ready 
access to essential evidence that documents the rights of American citizens, 
the actions of federal officials, and the national experience. 



Building for the Future 

Locations and 
Hours 



Washington. DC. 
Facilities 



Regional Archives 



The Visitor's Gallery 

□ Online Exhibit Hall: selected NARA exhibitions 

□ Gift Shop and Bookstore: publications and merchandise relating to 
NARA and its holdings 

□ The Digital Classroom : ideas, programs, and publications for the 
teacher 

□ Public Programs : conferences, training programs, lectures, films, 
tours, and other public events 



Records Centers 



Presidential Libraries 



Looking for Information in the National 
Archives 



Federal Register 



Genealogy and Individuals 



NARA Gopher 
What's New 



The Genealogy Page provides general information on NARA's resources on 
individuals, such as veterans, as well as guides to the use of NARA 
holdings. There are only a very few genealogical records currently available 
online . 



Highlights 



Historical Records of Government Agencies 




NARA makes available to the public the historically valuable records of the 
three branches of federal government: executive (including the President) . 
legislative , and judicial. National Archives holdings include textual; 
audiovisual : cartographic and architectural : and electronic records. 

The Guide to Federal Records in the National Archives of the United States 
provides general information about holdings for all three branches. You can 
also search for information about the National Archives in the following 
databases: 
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National Archives and Records Administration 



http://www.nara.gov/ 




Featured Document: 
A Letter from Jackie 
Robinson 




Records Management 
Workshops 



More Highlights 

□ Milo Rvan 
Phonoarchive 



INAKA Arcmvai information Locator yNAlLj 
John F. Kennedy Assassination Records Collection 

Current Government Information 



NARA publishes the Federal Register , the daily official record of the Federal 
government, the U.S. Government Manual , the Privacy, Act Issuances 
Compilation , the Public Laws . and the Code of Federal Regulations. The 
Government Information Locator Service (GILS) is a searchable database of 
information about NARA information resources with links to other sources 
of government information. 

NARA also coordinates the Electoral College process for choosing the 
President and Vice President of the United States. 

Professional Services 



Records management Archival management 

Grants NARA Library 

Preservation Internet Resources 

Employment and Internships 



□ Research and 
National History 
Day 



National Archives and Records Administration 
URL: http://www.nara.gov/ 

Questions. Comments, or QonwlaintsZ 
Last updated: April 10, 1997 
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Summary Report for www.nara.gov 



Hierarchy Pages Images Media Gateways Internet Back to Reports Help 

Error Report Duplicates Offsite InLinks Unexplored Back to Main Index 
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National Institutes of Health (NIH) 



http://www.nih.gov/ 




Welcome - News - Health - Grants - Science - Institutes - Employees 
Can't find it? Try using the NIH Search Eng ine. 

What’s New -- The latest information on major additions to the central NIH 
home page. 




Welcome to NIH 

An overview and introduction to NIH including an employee directory and 
maps of the NIH "campus" in Bethesda, Maryland. 




News and Events 

The NIH Calendar of Events, press releases, special reports, and employment 
information. 




Health Information 

A selection of some NIH health resources such as CancerNet, A ID S 
information, Clinical Alerts, the Women's Health Initiative and the NIH 
Information Index (a subject-word guide to diseases and conditions under 
investigation at NIH). 




Grants and Contracts 

Information on NIH's extramural research and training programs including 
NIH's funding opportunities (with application kits), grant policy, and award 
data that includes access to the CRISP database. 




Scientific Resources 

Intramural (on-site) research news and information including special interest 
groups, on-line library catalogs and journals, research training information, 
NIH research labs on the web and computer and network support for NIH 
scientists. 




Institutes and Offices 

The NIH home page is only the beginning. Here you will find links to the 
many individual organizations that collectively make-up the NIH. Some of 
these sites provide full-text consumer health publications. 
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http://www.nih.gov/ 



Information for Employees 

Work place information including emergency services, assistance for persons 
with disabilities, parking, facilities management, telecommunications, 
administrative resources such as the NIH manual chapters and the NIH 
WWW guidelines, employment information, and other services such as the 
NIH Federal Credit Union. 



We welcome your questions and comments; however, you may want to try our NIH Search 
Engine before writing. Please send general questions and comments about NIH and its 
medical research programs to the Office of Communications . Technical questions about this 
Web site can be addressed to the computer support staff . 



National Institutes of Health (NIH) 
Bethesda, Maryland 20892 
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St AT • USA 







Mostly Text Version 




STAT-US A/Internet 

□ What's New! 

□ How to Subscribe 

□ Daily Economic News 

□ Frequently Requested Statistical Releases 

□ Information By Subject 

□ Export & International Trade 

□ Domestic Economic News & Statistical Series 

□ Business Leads & Procurement Opportunities 

□ STAT-USA Databases 

□ Selected Publications of Interest 



□ the Newsletter 



A Service of the U.S. Department of Commerce 
Last Modified: April 03, 1997 I stat-usa@doc.gov I U.S. Department of Commerce 
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1. B. WEB SERVER LOG FILE ANALYSIS 
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Analysis of Stat-USA Stats - Access Log 



fIIe:///A|/L_SUSA-T.HTM 



Jeffrey H. Rubin 
March 20, 1997 



Analysis of Stat-USA Access Log: 
January 13, 1997 - January 20, 1997 

Path analysis of specific users 



Comparisons 

1* Number of hits vs. number of accesses 
2- Number of United S tates hits vs. outside countries 
3. Path analysis of specific users 

Analysis: 

Understanding the path (thread) in which a user navigates through a site can unfold multiple uses, such as clock 

TT<5 P A ge ’ e5Ut page ;u and m ° re ( . Tllis “Ctfop wiU allow you to click on actual IP addresses or domain names that 
hit the Stat-USA server over the week of our analysis. After you click on your user you wilt be presented with the comnlete 
thread of that users visit throught the Stat-USA site. This information will tell you what page the user entered the Stat-USA 

she on, what pages they saw while on the site, and finally what page they exited the site on. At the bottom of the malysis you 
will see the tune that each user spent on the Stat-USA site. analysis you 

Below is a list of a few randomly picked users to be used for path analysis, this should not crash your browser and it 
is intented to give a few examples of the usefulness of this type of analysis. **PLEASE NOTE** When you click on a user it 
may take a couple of minutes for the program to run and show you the results, please be patient. 
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413 


0.16503232715561 


198.83. 13, 45 


407 


0.16263476308071 


[1^239,70.21 


405 


0.161835575055743 


jgate.mckinsev.com 


404 


0.16143598104326 


biogen-bh.bioeen.com 


403 


0.161036387030777 


mfl07.logiccontrol.es 


400 


0.159837604993327 


www-e4.proxv.aol.com 


396 


0.158239228943394 


marketl.cci.cubic.com 


392 


0.15664085289346 


hostl4.cyberg8t.com 


391 


0.156241258880977 


gatewav5.ev.com 


390 


0.155841664868494 


wehcacheO Lu.washineton.edu 


390 


0.155841664868494 


toutainl.bschool.unc.edu 


385 


0.153843694806077 


|bastion.fdic.gov 


384 


0.153444100793594 


www-am3.proxv.aol.cpm 


374 


0.149448160668761 


www-ar3.proxv.aol.com 


356 


0.142255468444061 
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www.stat-usa.gov 

03.19.97 



Number of unique ip addresses: 14151 
How many arrive at initial page: 11763 

Percentage that arrive at initial page (# of initial / # of accesses): 8.323603710701 17 



Major Domain Statistics 


Domain type 


Total Hits 


Percentage of 
Total Hits 


Total Accesses 


Percentage of 
Total Accesses 


com 


78688 


31.4432536542872 


53964 


38.1854076888785 


edu 


34073 


13.6153667873441 


22388 


1 5. 84 1 94847 19185 


gov 


6379 


2.54901020563108 


3884 


2.74835304024172 


mil 


403 


0.161036387030777 


204 


0.144352219415373 


net 


38306 


15.3068482421859 


25298 


17.9010904253437 


o'rg 


3194 


1.27630327587171 


2124 


1.50296134332477 


United Arab Emirates 


146 


0.0583407258225643 


112 


0.0792521988947149 


Argentina 


169 


0.0675313881096806 


£>. 1 


0.0573163224149277 


Austria 


105 


0.0419573713107483 


65 


0.0459945797156827 


Australia 


816 


0.326068714186387 


500 | 


0.353804459351406 j 


Belgium 


242 


0.0967017510209627 


143 


0.101188075374502 


Bulgaria 


19 


0.00759228623718302 


10 


0.00707608918702811 


Bahrain 


8 


0.00319675209986654 


4 


0.00283043567481125 


Bermuda 


22 


0.00879106827463297 


16 


0.011321742699245 


Brunei Darussalam 


1 


0.000399594012483317 


1 


0.000707608918702811 


Bolivia 


19 


0.00759228623718302 


14 


0.00990652486183936 


Brazil 


408 


0.163034357093193 


261 


0.184685927781434 


Bahamas 


485 


0.193803096054409 


271 


0.191762016968462 


Canada 


5259 


2.10146491 164976 


3572 ! 


2.52757905760644 


Switzerland 


106 ; 


0.0423569653232316 j 


40 


0.0283043567481125 | 


| Chile ! 


160 


0.0639350419973307 ! 

■ - — „ .. ... ... i 


116 j 


0.0820826345695261 j 


1 China 


185 j 


0.0739248923094136 j 


121 


0.0856206791630402 j 


Colombia 


85 j 


0.0339654910610819 j 


41 


0.0290 1 1 9656668 1 53 j 


Costa Rica 


75 | 


0.0299695509362488 1 


40 1 0.028304356748 11 25 | 


Czech Republic 


1 19 j 


0.0475516874855147 j 


67 j 0.0474097975530884 j 


Germany j 


422 


0.16862867326796 


250 j 


0.176902229675703 


: Denmark j 


215 |0.0859127 12683913 1 j|l30 j|0.09 1989 15943 13655 
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Dominica 


4 ||0.00 159837604993327 


1 3 


i 0.002 122826756 10843 ! 


Dominican Republic 


106 ! 0.04235696532323 16 


93 


0.0658076294393615 i 


Estonia 


2 


0.000799188024966634 




0.00070760891870281 1 i 


Egypt 


21 


0.00839147426214966 


13 


0.00919891594313655 


Spain 


1044 


0.417176149032583 


445 


0.314885968822751 


Finland 


386 


0.15424328881856 


217 


0.15355113535851 


France 


844 


0.337257346535919 


545 


0.385646860693032 


Greece 


149 


0.0595395078600142 


133 


0.0941119861874739 


Guatemala 


46 


0.0183813245742326 


27 


0.0191054408049759 


Hong Kong 


165 


0.0659330120597473 


106 


0.075006545382498 


Croatia (Hrvatska) 


234 


0.0935049989210962 


189 


0.133738085634831 


Hungary 


95 


0.0379614311859151 


61 


0.0431641440408715 


Indonesia 


65 


0.02597361081 14156 


35 


0.0247663121545984 


Ireland 


567 


0.226569805078041 


543 


0.384231642855627 


Israel 


887 


0.354439889072702 


624 


0441547965270554 j 


India 


578 


0.230965339215357 


569 


0.4026294747419 


Iceland 


6 


0.0023975640748999 


4 


0.00283043567481125 


Italy 


858 


0.342851662710686 


584 


0.413243608522442 


Jamaica 


4 


0.00159837604993327 


2 


0.00141521783740562 


Jordan 


172 | 


0.0687301701471305 


147 


0.104018511049313 ! 


Japan 


1895 


0.757230653655886 


1014 


0.717515443564651 j 


j Korea (South) 


409 


0.163433951105677 


213 


0.150720699683699 j 


Kuwait 


273 


0.109089165407946 


244 


0.172656576163486 | 


Saint Lucia ' 


9 


0.00359634611234985 


6 


0.00424565351221687 


Liechtenstein 


47 


0.0187809185867159 


34 j 


0.0240587032358956 j 


Sri Lanka 


43 


0.0171825425367826 


33 


0.0233510943171928 


Lithuania 


6 


0.0023975640748999 


2 


0.00141521783740562 


Luxembourg 


1 


0.000399594012483317 


1 


0.000707608918702811 


Latvia 


57 | 


0.0227768587115491 


24 i 


0.0169826140488675 j 


Mexico 


448 | 


0.1790181 17592526 


305 | 


0.215820720204357 I 


Malaysia 


12) 


0.0483508755104814 


58 j 


0.0410413172847631 j 


Nicaragua | 


22 j 


0.00879106827463297 | 


11 i 


0.00778369810573092 j 


Netherlands | 


668 | 


0.266928800338856 i 


381 j 


0.269598998025771 


Norway 


415 j 


0. 1658315 15180577 | 


310 ! 


0.219358764797872 j 


New Zealand (Aotearoa)l 


M J 


0.033 166303036 1 1 53 j 


37 


0.026181529992004 


Peru | 


100 


0.03995940)2483317 j 


75 


0.0530706689027 109 


Philippines 


10 


0.00399594012483317 1 


4 


0.00283043567481 125 


(Pakistan 


60 |0.023975640748999 :[25 | 


0.0176902229675703 
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| Poland 


159 


|0.0635354479848474 


75 


j 0.0530706689027109 j 


| Portugal 


128 


|0.0511480335978646 


5JL __ 


1 0.0360880548538434 


Romania 


34 


0.0135861964244328 


19 


| 0.0134445694553534 


Russian Federation 


162 


0.0647342300222973 


92 


0.0651000205206586 


Sweden 


481 


0.192204720004475 


287 


0.203083759667707 


Singapore 


340 


0.135861964244328 


166 


0.117463080504667 


Slovenia 


24 


0.00959025629959961 


12 


0.00849130702443374 


Slovak Republic 


1 


0.000399594012483317 


1 


0.000707608918702811 


USSR (former) 


73 


0.0291703629112821 


42 


0.0297195745855181 


El Salvador 


84 


0.0335658970485986 


73 


0.0516554510653052 


Thailand 


509 


0.203393352354008 


311 


0.220066373716574 


Turkey 


203 


0.081 1 1758453411 33 


96 


0.0679304561954699 


Taiwan 


93 


0.0371622431609485 


59 


0.0417489262034659 


Ukraine 


39 


0.0155841664868494 


20 


0.0141521783740562 


United Kingdom 


1949 


0.778808730329985 


1206 


0.85337635595559 


United States 


1303 


0.520670998265762 


836 | 


0.59156105603555 


Uruguay 


6 


0.0023975640748999 


6 i 


0.00424565351221687 


Yugoslavia 


ii 1 


0.00439553413731649 


6 


0.00424565351221687 


South Africa 


217 


0.0867119007088798 | 


144 


0.101895684293205 


Other 


63429 


25.3458486178043 


16984 


12.0180298752485 | 


! ! 
| A UUU | 


250254 j 


100 


141321 


100 
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2. USER-BASED ASSESSMENTS 



A. Situated Proxies’ Assessments Evaluation Protocols: 1ST Webmasters 

Focus Group 

B. Intermediate Criteria 

C. Linkages Between User Questions And Criteria Chosen For Investigation 

D. Sign-Up Sheet To Participate In Assessment Of Federal Websites 

E. Researcher’s Instructions For Conducting Situated Analysis 

F. Demographics And Survey Form 

G. Assessment Criteria (Used) 

H. Assessment Criteria (Revised) 

I. Focus Group Probes 



2. A. SITUATED PROXIES’ ASSESSMENTS EVALUATION PROTOCOLS: 

1ST WEBMASTERS FOCUS GROUP 

The purpose of this focus group was to gather information from experienced Internet users to develop 
potential questions which can be used in an website assessment. The group consisted of six 
participants, the researcher and myself. 

The discussion began with a study team member describing the intention of the focus group. 
Participants then questioned whether we knew who the users of the website would be. Absence of 
this knowledge prompted difficulty in their ability to evaluate a website. Many felt that this was 
essential information to perform the task at hand. 

After informing participants that the users would not be known the researcher asked the group to 
come up with a list of criteria or smaller questions for each of seven pre-prepared questions. 

A listing of each of the pre-prepared questions followed by the focus groups responses proceeds. 
Note, responses are listed as possible smaller questions to answer pre-prepared questions. 

Qi Does the layout of the page present a logical organization of the information and services 
available? 

[it was suggested that the contents page is prone to have logical organization. The 
question arose however as to what page we are talking about or if we are talking about the 
entire site] 

• Does the site have some kind of navigation 

• Is the table of contents logical 

[someone mentioned that there can be different kinds of logical organization, linear or 
chronological] 

[another participant suggested that the first page is a splash screen which entices the user, 
and the next page is where the navigation of the site should begin] 

[the discussion was then directed specifically at the homepage] 

• Does it provide mechanisms to navigate through site 

• Does the layout help the user understand the overall structure of site 

• The first couple of pages should provide depth of site 

[someone commented on how this might be problematic. For example, how many links 
determines depth?] 

• Does the user know what the website is about from this page 

• Is the wording on the page easy to understand 

[the discussion then moved into what constitutes a good layout] 

• Is it one that the user can interface with, with little to no difficulty 

• Do things line up, is information separated (there should be defined categories) 

• Does it fit different platforms (different screen sizes) 

• Is the navigation clearly designed 

• Is there built in navigation or does the user have to use the browser to navigate 

• Is user able to find what they are looking for 
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* Does it attract people 



Q2 Does the website design seem cluttered? 

[again, participants questioned whether we were talking about one page or multiple pages. 
The decision was to assume one page] 

• Is the page too wordy 

• Are there too many graphics 

• Are there too many links, options for users 

• It is to cluttered, does the user have to shift through many pages 

[someone suggested that it might be helpful to look at how people evaluate the design of 
print material for possible guidance] 

[at this point the discussion focused on the website as a whole] 

• Is there consistency throughout (visual cues and navigation is the same) 

• Does navigation get the user lost or is the user unable to find navigation 

• Is the hierarchy clearly defined 

• Is there clear definition of where the user is in the site, the Web is relational not just 
hierarchical 

• Does every page have a name to inform the user of what site he/she is on 
Q3 Is the design consistent throughout the site? 

• Things should be kept in the same areas and the same size. For example, navigation 
should always be the same and in same place. Banners should also always be the 
same size 

• Is the content the same throughout 

[someone suggested that the user should have the ability to download the entire document 
so that person does not have to go back and forth through all the different pages] 

• If it is one long document does if offer hotlinks to jump to desired areas 

• The background should have a consistent metaphor throughout 

Q4 Is help information needed in order to navigate this website? 

[one participant suggested that if the user needs help then something is wrong with the 
site] 

[it was also suggested that there is a certain basic knowledge that users of the Web should 
posses (e.g., blue text is a hyperlink and will take the user somewhere else, forward, 
backward). If every site was to discuss these fundamental things then sites could get 
frustrating] 

[one person commented that a map of the entire site would be nice. This would allow the 
user to rise above the maze to view where they were] 

[another person mentioned that a hierarchy of menu options could also be used instead of a 
map which would achieve the same thing] 

• Is text to small to read 
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• Does site offer help about its particular layout or icons 

[one suggestions was that the site could offer a list of frequently asked questions] 

• If site has a search engine does it support it with help instructions 

[it was also mentioned that contact information on a site is a nice feature] 

• Are there ways to get more information 

Q5 To what degree is skill in navigating the Internet needed for navigating this website? 

[someone pointed out that if the user has to use the browser then there is no navigational 
tools on the site] 

• Are backward and forward buttons consistent with Internet standards 

• Is the terminology consistent with where the user actually ends up 

• Are there descriptive words which describe where a link will take the user 

[it was discussed that a site could offer a button that offers navigational help as well as 
Internet basics and could even be divided up by level of expertise (ex. novice or advance)] 

• What level of knowledge does page assume 
Q6 Do the graphics guide the user through the website? 

• Is the graphic clearly defined as a link or just an image 

• Are the graphics consistent throughout 

• Are the tools consistent same throughout 

Q7 Does the help information explain how to navigate within the website? 

• Does it explain what the icons or images mean 

• Does each time the user use a help tool it performs the same 

• Does the site explain what underlined text means 

• Did the help information answer the user's question 

• Did the help information explain the site's structure 

• Did the help information explain the site's objective 

• Did the help information explain the site’s navigation 

[it was suggested by a participant that this question was closely related to Q4 or could 
even serve as a sub-question to Q4] 



2. B. INTERMEDIATE CRITERIA 



Criteria For The Evaluation Of Websites 
I. INFORMATION CONTENT CRITERIA 

This section evaluates the substantive aspects of the website. 

Orientation to Website 

• A website overview is provided. 

• The scope of website is clearly stated. 

• The services and information provided at the website are described. 

• Instructions on the use of the website are provided. 

• The purpose/mission should be appropriate to the targeted audience(s). 

• A liability statement warning the user of information provided through the links is provided ( e.g ., 
access by children). 

• Copyright statements are provided if necessary. 



Content 

• The content of the homepage should match the purpose/mission. 

• The content and links match the needs of the expected audience. 

• The website includes only necessary and useful information. 

• Content coverage does not overlap. 

• The amount of information is significant and not overwhelming and is balanced throughout the 
website. 

• Full text references or other resources are available. 

• The content has rich and unique qualities that inspire users to visit the site regularly, 

• The content is written in a clear and consistent language style that matches the expected 
audience. 

• Avoids jargon, humor, condescension, accusation, and chit chat. 

• Uses a positive and professional tone. 

• Language does not show bias. 

• Content is organized logically throughout the website and by anticipated user need. 

Currency 



• Address of contact person and last update information appears at the bottom of pages with 
substantive content. 

• Pages have been updated in the past three months. 

• The content provided by the website creators is up-to-date. 

Bibliographic Control 

• Headings are clearly phrased, descriptive and understandable. 

• Each screen is titled clearly. 

• If the headings cannot be completely descriptive, coherent and concise, descriptions follow. 

• The information that is provided through the link matches the headings and descriptions. 

• Terminology and layout are consistent within the headings throughout the website. 
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Services (if provided) 



• Provision of services is different than provision of information resources, (e.g., leaving a 
question to be answered by the reference staff is a service). 

• Are the services open to everyone on the Internet or do portions require fees? 

• Services meet the needs of the user. 

• Services are fully operational. 

Accuracy 

• Statement of document/website status is provided (e.g., if in progress will note, "under 
construction”). 

• Reference or sources of information cited are accurate. 

• Typing, spelling, and grammar errors and other inconsistencies are absent. 

Privacy 



• Degree to which the site provides users with explicit policy on how users’ privacy rights are 
protected. 

• Degree to which the site provides users with information about making public site-use 
information, repackaging, or selling such information to others. 

• Can the user exchange encrypted information with the site? 

II. EASE-OF-USE CRITERIA 



This section evaluates physical movement through the website. 

Quality of Links 

• There are no dead-end links. 

• Temporary forwarding addresses do not qualify as good links. 

• "What's new" section provided for new links (good for frequent users). 

• Shortcut links are possible for frequent users. 

• Warning statements are provided if link will lead to large document or image. 

• Indication of restricted access for a link is provided. 

• Links are provided to mentioned documents. 

• Speed. 

• Minimal use of large graphics and bright colors so that document loading is relatively fast. 
Feedback Mechanisms 

• On the homepage and other significant pages there is a contact person: name, address and email 
address of the responsible person or entity; for example, <webadmin@library.org>. 

• Feedback links are fully operational. 



Accessibility 



• There is an awareness of the existence of this website through search engines or other publicity. 

• Full name of website organization is provided in the title, heading, document address, graphical 
link and/or URL in order that the source be recoverable. 

• It is usually possible to reach the site. It is not frequently overloaded. 

• The URL is not likely to be confused or mistyped. 
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Design 



• Format is appropriate to subject matter and functionality. A good design directs users toward 
information rather than away from it. 

• The screens are uncluttered. 

• The format is consistent throughout the website. 

• The homepage for the website is short and simple. 

• The website is written in standard HTML language. The site is consistent when accessed via 
different browsers (text and graphic). 

• Graphics and color lead the user through the information appropriately. 

• Monster graphics are not used. 

• Many little graphics are avoided. 

• Use of bold, italics, blinking and other attention-getting devices is limited. 

• User has the option of turning off the automatic loading of graphics. 



Navigability 



• Essential instructions appear before links requiring user interaction (e.g., email). 

• Navigation options are distinct and spelled out. 

• Minimal user skills are required. 

• All the parts work. 

• Interactions are secured if they involve private information. 

• Links are provided to return to website homepage on all supporting pages. 

• Links are provided to assist navigation ( e.g ., "return to top," "return to previous page"). 
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2. C. LINKAGES BETWEEN USER QUESTIONS AND CRITERIA CHOSEN 
FOR INVESTIGATION: 

The content and the links match the needs of the expected audience. 

To what extent does the content fulfill your needs or interests? 

Rate the quality of the site. 

The overall content of the site matched my expectations. 

Using this site would enable me to accomplish my desired tasks more quickly. 

The site provided the precise information I needed. 

The information on the site was relevant to me.. 

The scope of the website is clearly stated. 

The site clearly indicated its scope and content. 

I understand the purpose of this site. 

The services and information provided at the website are described. 

The site clearly described its services and information. 

Content is organized logically throughout the website and by user’s needs. 

The site organized its information in a way that was easy for me to understand.. 

Language does not show bias. 

To what extent does the content avoid bias? 

The amount of information is significant and not overwhelming and is balanced throughout the website. 
To what extent does the site provide sufficient information? 

Headers are clearly phrased, descriptive and understandable. 

To what extent are the headers understandable? 

The information that is provided through the links matches the headings and descriptions. 

To what extent do the headers accurately reflect the content they describe? 

Format is appropriate to the subject matter and functionality. 

Rate the overall site deign. 
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The format is consistent throughout the website. 

To what extent is the design consistent throughout the site? 

Graphics and color are used to lead the user through the information appropriately. 

To what extent is the use of graphics appropriate for the site? 

To what extent are any of the design elements distracting? 

The graphics helped me find my way around the site. 

Minimal user skills are required. 

It was easy to move to where I wanted to go. 

It was easy to navigate around the site. 

I would find it easy to find what I want at this site. 

It would be easy for me to become skillful at using this site. 

The site was easy to use. 

Teaching a beginner to use this site would be easy. 

It was easy to provide feedback to the site. 

Navigation options are distinct and spelled out. 

It is easy to understand what the navigation icons mean. 

Feedback mechanism. 

It was easy to provide feedback to the site. 




2. D. SIGN-UP SHEET TO PARTICIPATE IN ASSESSMENT OF 
FEDERAL WEBSITES 



Name: Email: 

Phone: 

Availability - Please indicate exact days and times you are available. 
Week of Nov. 25-29 



Week of Dec. 2-6 



Week of Dec. 10- 13 



Listed below are the available websites for user-based assessment (Please circle which site(s) you 
would be interested in visiting) 

1. Department of Commerce (Stat-USA) 

2. Department of Defense 

3. General Services Administration 

4. National Archives and Records Administration 

5. FedWorld Information Network 

6. National Aeronautics and Space Administration 

7. Bureau of the Census 

8. National Institutes of Health 

9. Federal Communications Commission 

For more information about these sites see the back of this form. 



0 
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2. E. RESEARCHER’S INSTRUCTIONS FOR CONDUCTING 
SITUATED ANALYSIS. 



1 . Set up Web browser and bookmark a warm-up site and the research site. 

2. Thank the participant for coming, restate the purpose of the study and provide confidentiality 
statement, and ask if there are any questions. — See A under participant instructions below. 

3. Have participant fill out demographic data and give them an opportunity to glance over the rest of 
the survey form. 

4. Provide instructions for the warm-up exercise. — See B under participant instructions below. 

5. Conduct warm-up exercise - approximately five minutes. 

6. Give the participant a chance to ask any question that may have arisen during the warm-up and 
reiterate instruction for the test. - See C under participant instructions below. 

7. After completion of test, spend a few minutes with the participant (still on tape) answering 
participant questions and asking the participant questions that may have occurred to you during the 
session. 

8. Ask the participant to complete the questionnaire. 

9. Thank the participant for their time and remind them to attend the focus group session - 
Wednesday, December 1 1th — 1ST Conference Room. 

Participant Instructions. 

Ask participants to excuse you for reading these instruction verbatim. While this may be a bit 
awkward, it will ensure that we do not mix up any of the instructions. 

A. Thank you for agreeing to participate in this study. The purpose of this study is to identify and 
test assessment criteria and techniques that can be used to evaluate federal websites. Your 
participation is extremely helpful as it allows us to observe how an “interested” user interacts with a 
website. 

We are going to use a combination of techniques to collect data during this session. We have a 
structured instrument for you to complete after the session and will be taping you during your Web 
visit. 

We want you to understand that we are evaluating the website, not you, or your use of the website. 

In this regard, it is not really possible for you to make a mistake or do something wrong. To the 
extent possible, we hope that with a minor exception that we will explain, that you conduct your visit 
just as you normally would. 

Although we don’t anticipate confidentiality being an issue, be assured that the tapes and survey 
forms will only be accessed by the survey team and will be destroyed after the study is completed. 
Transcripts made from tapes and study findings may be retained but will not include personal 
identifiers. 

Are you still willing to participate in this study? 
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Do you have any questions so far? 

Now we would like you to fill out the demographic portion of the survey, then take some time to read 
the rest of the questionnaire so that you will understand the types of questions that we will be asking 
you to answer. 

<pause> 

Do you have any questions? 

B. We are going to be using the “think aloud” method for collecting data during your Web visit. The 
method is fairly simple, but takes some getting used to as you will be asked to verbalize your thoughts 
as you work your way through the Web site. For many people, verbalizing their “stream of 
consciousness” is a bit unnatural. Therefore, we will have a short warm-up session to give you an 
opportunity to get used to the technique. 

During the practice and your chosen site evaluation, we actually want you to perform two tasks at 
once. First, you will be searching the website for information relevant to your information need. 
Second, you will evaluate the website as you conduct your search. 

The Think Aloud technique requires you to say out loud what enters your mind as you navigate the 
website. In addition to prompting you to continue talking, we will occasionally point to the banner at 
the top of the screen to remind you to continue to make evaluative comments. However, you do not 
need to limit your comments to the listed criteria. If any particular feature about the site strikes you 
as being positive or negative just say it. You don’t need to explain why you think so — just try to say 
everything that comes into the mind. 

You can talk to the researcher and ask questions, although there may be some questions that we will 
not answer in order to minimize our biasing your responses. We will prompt you to keep talking if 
you pause for more than a coupie of seconds. Don’t worry if off the thoughts in your mind seem 
ridiculous, or even if you are thinking that this exercise is ridiculous — go ahead and say it! The trick 
is to pretty much let us fade into the background and just keep talking to yourself. 

Furthermore, if you become “stuck” in one section of the website, we may urge you to move on. 

We would like to give you a brief demonstration of a “think out loud” based site evaluation in order 
to make you feel more comfortable talking and to give you some idea of what we expect. 

<give demonstration> 

Any questions? 

Ok, now its your turn! Remember, this is just for practice! 

<Connect to the test site and begin> 

<pause> About 5 minutes no longer than 10. Less time is OK if the participant is comfortable with 
the technique. 

OK, any questions? 

Do you want another warm-up session? 
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C. Let’s start the session. You do not have a set time that you need to spend visiting this site. We 
would like your visit to last at least 20 minutes and we will cut you off at the end of 45 minutes in 
order to get ready for the next participant. You were selected because you had some reason for 
visiting this site, school, work or personal. 

Think about the purpose of your visit for a few seconds before starting your session. 

<pause> What was your purpose? 

Remember, we want you to just say whatever comes into your mind as you are navigating this site. 
However, we are particularly interested in hearing your evaluative thoughts and will occasionally 
point to the banner at the top of the screen to prompt you to make evaluative comments. When we 
point to the banner, it is not to prompt you to make a comment about a specific evaluative criterion. 
Rather, it is just to remind you to make “evaluative-like” comments. Feel free to comment on things 
other than the criteria listed on the banner. 

We do ask that you primarily spend your time navigating the site to locate information rather then 
spending a great amount of time reading or downloading a particular document. Of course, looking 
over identified text to verify that you are interested in it or not is certainly appropriate as we do ask 
some questions concerning the quality of the website’s content. If you fmd something that you are 
interested in, just bookmark the page and we can come back and retrieve it after the session is 
completed. 

To summarize: During the session we will prompt you to keep talking, we may point at the banner at 
the top of the screen to remind you to make evaluative comments, and we may urge you to continue 
exploring the site if you become bogged down in a particular section. 

Any final questions? 

Please proceed. 
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2. F. DEMOGRAPHICS AND SURVEY FORM 



OCLC FEDERAL WEBSITE SITUATED USER ASSESSMENT SURVEY 

12 / 1/96 



Name 

Age 

Major/Concentration 

Estimated Number of web pages Designed 

Number of Months/Years navigating the WWW 

How would you rate your skill in navigating the WWW? 
low very high 

1.2 3 4 5 



How would you rate your experience using electronic sources of Federal Government Information? 

low very high 

1 2 3 4 5 

How would you rate your experience evaluating the design of websites? 

low very high 

1 2 3 4 5 



How would you rate your experience evaluating the content of websites? 

low very high 

12 3 4 5 



in 
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2. G. ASSESSMENT CRITERIA (USED) 






A. Quality Issues 

1 . To what extent is the content balanced or unbiased? 


Poor 

1 


2 


3 


4 


Excellent 

5 


2. To what extent does the content fulfill your needs or 


1 


2 


3 


4 


5 


interests? 


3. To what extent does the site provide in-depth information? 


1 


2 


3 


4 


5 


4. Rate the overall quality of the site 


1 


2 


3 


4 


5 


5. I would recommend this site to a colleague. 


1 


2 


3 


4 


5 


6. To what extent do you trust the information at this site? 


1 


2 


3 


4 


5 


B. Usefulness Issues 

1 . The content of the site matched my expectations. 


Disagree 
1 2 


3 


4 


Agree 

5 


2. The site clearly indicated its scope and content. 


1 


2 


3 


4 


5 


3. The site clearly described its services and information. 


1 


2 


3 


4 


5 


4. The site organized its information in a way that was easy 


1 


2 


3 


4 


5 


for me to understand. 


5. Using this site would enable me to accomplish my desired 


1 


2 


3 


4 


5 


tasks more quickly. 


6. The site provided the precise information I needed. 


1 


2 


3 


4 


5 


7. The information on the site was relevant to me. 


1 


2 


3 


4 


5 


8. I understand the purpose of this site 


1 


2 


3 


4 


5 




EASE OF USE : 



A. Presentation Issues 


Poor 








Excellent 


1. To what extent are the headers understandable? 


1 


2 


3 


4 


5 


2. To what extent do the headers accurately reflect the 


1 


2 


3 


4 


5 


content they describe? 












3. Rate the overall site design 


1 


2 


3 


4 


5 




Not At AH 






A Great Deal 



4. To what extent is the design consistent throughout the site? 


1 


2 


3 


4 


5 


5. To what extent is the use of graphics appropriate for the 
site? 


1 


2 


3 


4 


5 


6. To what extent are any of the design elements distracting? 


1 


2 


3 


4 


5 



B. Navigation Issues 


Disagree 






Agree 


1 . The graphics helped me find my way around the site 


1 


2 


3 


4 


5 


2. It was easy to move to where I wanted to go. 


1 


2 


3 


4 


5 


3. It is easy to understand what the navigation icons mean 


1 


2 


3 


4 


5 


4. It is easy to find what I want at this site. 


1 


2 


3 


4 


5 


C. General 


Disagree 






Agree 


1 . It would be easy for me to become skillful at using this site. 


1 


2 


3 


4 


5 


2. The site was easy to use. 


1 


2 


3 


4 


5 


3. Teaching a beginner to use this site would be easy. 


I 


2 


3 


4 


5 


4. It was easy to provide feedback to this site 


1 


2 


3 


4 


5 




BEST COPY AVAILABLE 
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2. H. ASSESSMENT CRITERIA (REVISED) 



CONTENT 



A. Quality Issues 



Poor 



Excellent N/A 



1 . To what extent did the site present different types or sources 1 2 3 4 5 

of information about your topic, thereby avoiding bias? 

2. To what extent does the content fulfill your needs? 1 2 3 4 5 

3. To what extent does the site provide in-depth information? 1 2 3 4 5 

4. Rate the overall quality of the site 1 2 3 4 5 

5. I would recommend this site to a colleague. 1 2 3 4 5 

6. To what extent do you think the information provided by 1 2 3 4 5 

this 

site is true or correct? 

7. To what extent does information quality vary within the 
site? 



B. Usefulness Issues 



Disagree 



Agree N/A 



1. The content of the site matched my expectations. 1 2 3 4 5 

2. The site clearly indicated its scope and content. 1 2 3 4 5 

3. The site clearly described its services and information. 1 2 3 4 5 

4. The site organized its information in a way that was easy 1 2 3 4 5 

for me to understand. 

5. Using this site would enable me to accomplish my desired 12 3 4 5 

tasks more quickly. 

6. The site provided the precise information I needed. 1 2 3 4 5 

7. The information on the site was relevant to me. 1 2 3 4 5 

8. I understand the purpose of this site 1 2 3 4 5 

C. General Questions 

1 . What aspect of the site did you like the most? 

2. WTiat aspect of the site did you like the least? 

3. Please make one suggestion for improving the website. 



O 

ERLC 
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A. Presentation Issues 



Poor Excellent N/A 



1 . To what extent are the text headers understandable? 

2. To what extent do the headers accurately reflect the 
content they describe? 

3. Rate the overall site design 

4. To what degree was the information at the site located where 
indicated by the headers? 

5. To what extent is the design consistent throughout the site? 

6. To what extent is the use of graphics appropriate for the 
site? 

7. To what extent are any of the visual elements distracting? 

B. Navigation Issues 

1 . The graphics helped me find my way around the site. 

2. It was easy to move to where I wanted to go. 

3. It is easy to understand what the navigation icons mean. 

4. It is easy to find what I want without using a search engine. 

5. Chance played a big part in my finding the information I 
wanted. 

C. General 

1 . It would be easy for me to become skillful at using this site. 

2. The site was easy to use. 

3. Teaching a beginner to use this site would be easy. 

4. It was easy to provide feedback to this site 



1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

Not At All Great Deal N/A 



1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 



Disagree 




Agree N/A 


1 2 


3 


4 


5 


1 2 


3 


4 


5 


1 2 


3 


4 


5 


1 2 


3 


4 


5 



Disagree 




Agree N/A 


1 2 


3 


4 


5 


1 2 


3 


4 


5 


1 2 


3 


4 


5 


1 2 


3 


4 


5 



2. I. FOCUS GROUP PROBES 



What constitutes good navigability? 

How legitimate is the information on federal websites? 

How are searching skills in Web environments different from search engines? 

Do you feel that you are able to tell whether something you did or did not like was due to the website, 
the browser, the local network, etc .1 

Are there criteria we should have included but did not? 

Are there criteria that seemed inappropriate or unclear? 

Were you comfortable with the Think Aloud technique? 

How did being videotaped and observed affect your assessment? 

Did you feel that your interest (situatedness) in the site’s content affected your assessment? 

Was the website you assessed better than, worse than, or about what you expected? 

Is method useful to agencies? 

Are criteria transferable to other sites? 

Is sample appropriate? 





3. DESIGN-BASED ASSESSMENTS 



A. Technical Assessment Criteria -- Summary Of All Four Websites* 

B. Technical Assessment Criteria -- NIH Website 

C. Technical Assessment Criteria — NARA Website 

D. Technical Assessment Criteria — Stat-USA Website 

E. Policy Assessment Criteria -- Summary Of All Four Websites 

F. Policy Assessment Criteria -- NIH Website 

G. Policy Assessment Criteria — NARA Website 

H. Policy Assessment Criteria - Stat-USA Website 
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3. A. TECHNICAL ASSESSMENT CRITERIA - Summary of All Four Websites* 



Technical Criteria 
Content: Quality Issues 

1 . Do hyperlink headings and descriptions match the 
information found through the hotlinks? 

Usefulness Issues 

1 . Are descriptions given to further describe the headers? 

2. Are services offered on the site? 

3. Are the services functional? 

Currency Issues 

1 . Does the website include dates indicating when 
pages were last updated? 

2. How long since the last update? 

Ease of Use/Presentation Issues 

1 . Are the banners consistently designed? 

2. Are the banners consistently placed? 

3. Are the navigation bars consistently designed? 

4. Are the navigation bars consistently placed? 

Navigation Issues 

1 . Do hotlinks connect to gopher menus? 

2. Is any of the site restricted by passwords? 

3. Is the page longer than three screens? 

4. Do pages longer than three screens provide anchors? 

5. Does the site consistently differentiate between 
graphics and graphical links? 

6. Does the site have a consistent way of indicating 
text and graphical hotlinks? 

7. Do you need to use the browser to navigate? 

8. Does each page provide an indication of where 
you are in the site? 

9. Does each page indicate what site you are on? 

10. Does the site warn you when you take an outside link? 



Census 1 


NIH 2 


NASA 1 


STAT-USA 


5 


5 


5 


3 



1 


3 


3 


3 


4 


0 


5 


5 


4 


0 


4 


5 



5 


0 


4 


5 


10/01/96- 


Not 


09/13/96 


10/04/96 


02/12/97 


provided 


01/15/97 


02/06/97 


5 


5 


4 


5 


5 


5 


4 


5 


2 


5 


3 


5 


2 


5 


3 


5 



1 


2 


2 


0 


0 


0 


0 


3 


1 


2 


3 


0 


0 


0 


0 


0 


3 


0 


0 


2 


2 


3 


5 


5 


0 


0 


1 


0 


5 


5 


5 


5 


5 


5 


4 


5 


0 


0 


0 


0 
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Help Issues 



1 . Is there a person or address that can be reached to provide 

feedback or to ask questions? 2 

2. Are large documents broken up into subsections? 2 

3. Are indexes provided for large documents and subsections? 1 

4. Does the site provide a download option for large 

documents? 1 

5. Does the site warn you about file sizes? 1 

6. Is a text only version of the site available? 1 



5 

0 

0 

0 

0 

0 



5 5 

0 1 

0 0 



0 0 
1 0 
0 1 



1. United States Census Bureau <http://www.census.gov> 

2. National Institutes of Health <http://www.nih.gov> 

3. National Archives and Records Administration 
<http://www.nara.gov> 

4. Department of Commerce <http://www.stat-usa.gov> 



* The numbers 0-5 indicate the number of “YES” responses for each website, 
based on five webpages per each of the four websites assessed. Note that a zer 
does not necessarily mean “NO”; it could also indicate that the question is not 
applicable to a particular page. Refer to the detailed assessments for individual 
websites provided below. 




no 



3. B. TECHNICAL ASSESSMENT CRITERIA ~ NIH Website 



Technical Criteria Page I 1 

Content: Quality Issues 

1 . Do hyperlink headings and descriptions match the 

information found through the hotlinks? Y 

Usefulness Issues 

1 . Are descriptions given to further describe the headers? Y 

2. Are services offered on the site? N 

3. Are the services functional? N/A 

Currency Issues 

1 . Does the website include dates indicating when 



pages were last updated? N 

2. How long since the last update? N/A 

Ease of Use/Presentation Issues 

1 . Are the banners consistently designed? Y 6 

2. Are the banners consistently placed? Y 6 

3. Are the navigation bars consistently designed? Y 7 

4. Are the navigation bars consistently placed? Y 7 



Navigation Issues 



1 . Do hotlinks connect to gopher menus? N 

2. Is any of the site restricted by passwords? N 

3. Is the page longer than three screens? N 

4. Do pages longer than three screens provide anchors? N/A 

5. Does the site consistently differentiate between 

graphics and graphical links? N 8 

6. Does the site have a consistent way of indicating 

text and graphical hotlinks? Y 

7. Do you need to use the browser to navigate? N 

8. Does each page provide an indication of where 

you are in the site? Y 

9. Does each page indicate what site you are on? Y 

10. Does the site warn you when you take an outside link? N/A 



Page II 2 Page III 1 Page IV- Page V 



Y 



Y 

N 

N/A 



Y 



Y 

N 

N/A 



Y 



N 

N 

N/A 



Y 



N 

N 

N/A 



N 

N/A 



Y 6 

Y 6 

Y 7 

Y 7 



N 

N 

Y 
N 

N 8 

Y 
N 

Y 

Y 
N 9 



N 

N/A 



Y 6 

Y 6 

Y 7 

Y 7 



Y 
N 

Y 
N 

N 8 

Y 
N 

Y 

Y 
N 9 



N 

N/A 



Y 6 

Y 6 

Y 7 

Y 7 



Y 

N 

N 

N/A 

N 8 



N 

Y 

Y 
N 



N 

N/A 



Y 6 

Y 6 

Y 7 

Y 7 



N 

N 

N 

N/A 

N 8 



N 

Y 

Y 
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Help Issues 



1. Is there a person or address that can be reached to provide 

feedback or to ask questions? Y Y Y Y 



2. 


Are large documents broken up into subsections? 


N/A 


N/A 


N/A 


N/A 


J. 


Are indexes provided for large documents and subsections? 


N/A 


N/A 


N/A 


N/A 


4. 


Does the site provide a download option for large 












documents? 


N/A 


N/A 


N/A 


N/A 


5. 


Does the site warn you about file sizes? 


N/A 


N/A 


N/A 


N/A 


6. 


Is a text only version of the site available? 


N 


N 


N 


N 



1. National Institute of Health (NIH) - Homepage <http://www.nih.gov> 

2. National Institute of Health — What’s New <http://www.nih.gov/whatsnew.html> 

3. National Institute of Health — Health Information Index <http://www.nih.gov/health/> 

4. National Institute of Health - Scientific Resources <http://www.nih.gov/science/> 

5. National Institute of Health — Institutes, Centers, and Divisions <http://www.nih.gov/icd/> 

6. Banners were consistently designed and placed on the five pages assessed. 

7. The navigation bar is a list of the different sections of the site along the bottom of the page and is consistently placed. 

8. Of the five pages assessed there is consistency among graphics and graphical links. 

9. Indirectly in the description of the links there is mention of another site, but it is not clearly stated that the links are to other sites. 

10. This page is a list of many links with no further explanation about them. This is inconsistent with the other pages assessed. 

1 1. This page is a list of links to many institutes and centers with no indication that the links are to other site except indirectly in the 
name of the links. 




Y 

N/A 

N/A 

N/A 

N/A 

N 
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3. C. TECHNICAL ASSESSMENT CRITERIA - NARA Website 



Technical Criteria 
Content: Quality Issues 

1 . Do hyperlink headings and descriptions match the 
information found through the hotlinks? 

Usefulness Issues 

1. Are descriptions given to further describe the headers? 

2. Are services offered on the site? 

3. Are the services functional? 

Currency Issues 

1 . Does the website include dates indicating when 
pages were last updated? 

2. How long since the last update? 

Ease of Use/Presentation Issues 

1 . Are the banners consistently designed? 

2. Are the banners consistently placed? 

3. Are the navigation bars consistently designed? 

4. Are the navigation bars consistently placed? 

Navigation Issues 

1 . Do hotlinks connect to gopher menus? 

2. Is any of the site restricted by passwords? 

3. Is the page longer than three screens? 

4. Do pages longer than three screens provide anchors? 

5. Does the site consistently differentiate between 
graphics and graphical links? 

6. Does the site have a consistent way of indicating 
text and graphical hotlinks? 

7. Do you need to use the browser to navigate? 

8. Does each page provide an indication of where 
you are in the site? 

9. Does each page indicate what site you are on? 

10. Does the site warn you when you take an outside link? 



Pape I 1 


Page II 3 


Page III 1 Paae IV- 


Pace V 


Y 


Y 


Y Y 


Y 



N 6 


N 


Y 


Y 


N 


Y 7 


Y 


Y 


N 


Y 


Y 


Y 


Y 


N/A 


Y 



Y 


Y 


Y 


N/A 8 


Y 


01/15/97 


09/13/96 


01/06/97 




10/07/9 



Y 9 


Y 


Y 


N 10 


Y 


Y 


Y 


Y 


N/A 


Y 


N/A 11 


Y 12 


Y 12 


N 13 


Y 12 


N/A 


Y 


Y 


N/A 


Y 



Y 


Y 


N 


N 


N 


N 


N 


N 


N 


N 


Y 


N 


Y 


Y 


N 


N 


N/A 


N 


N 


N/A 


N 14 


N 14 


N 14 


N 14 


N 14 


Y 


Y 


Y 


Y 


Y 


N/A 15 


N 16 


N 16 


Y 


N 16 


Y 


Y 


Y 


Y 


Y 


Y 


Y 


Y 


N 17 


Y 


N/A 


N/A 


N/A 


N/A 


N/A 



0 
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Help Issues 



1. Is there a person or address that can be reached to provide 



feedback or to ask questions? 

2. Are large documents broken up into subsections? 

3. Are indexes provided for large documents and subsections? 

4. Does the site provide a download option for large 




Y 

N/A 

N/A 



Y 

N/A 

N/A 



Y 

N/A 

N/A 



Y 

N/A 

N/A 



N/A 

N/A 



documents? 

5. Does the site warn you about file sizes? 

6. Is a text only version of the site available? 



N 

N 

N 



N 

N 

N 



N 

N 

N 



N 

Y 

N 



N 

N ‘ 
N/A 



1. Homepage <http://www.nara.gov> 

2. What’s New at NARA <http://www.gov/nara/whatsnew.html> 

3. The Digital Classroom <http://www.nara.gov/nara/digital/classroom.html>> 

4. The Japanese Surrender Document <http://www.nara.gov/exhall/wwii/surrender.html> 

5. NARA and the Public <http://www.nara.gov/nara/whatis/public.html> 

6. Varies throughout page 

7. The site itself is a service? 

8. This page will always remain the same 

9. Homepage has unique banner; - 1 0. No banner at top of page; —11. No navigation of any kind 

12. Home button at bottom left; - 13. No home button at bottom left; - 14. No distinct differences on page; - 15. Evaluation of 

one page only 

16. Pages had home button at bottom left’ - 17. No header at top of page; - 18. Link reads, “Questions, Comments, or Complaints” 



BEST COPY AVAILABLE 




3 



ERJC 
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3. D. TECHNICAL ASSESSMENT CRITERIA - Stat-USA Website 



Technical Criteria 
Content: Quality Issues 

1 . Do hyperlink headings and descriptions match the 
information found through the hotlinks? 

Usefulness Issues 

1 . Are descriptions given to further describe the headers? 

2. Are services offered on the site? 

3. Are the services functional? 

Currency Issues 

1 . Does the website include dates indicating when 
pages were last updated? 

2, How long since the last update? 

Ease of Use/Presentation Issues 

1 . Are the banners consistently designed? 

2. Are the banners consistently placed? 

3. Are the navigation bars consistently designed? 

4. Are the navigation bars consistently placed? 

Navigation Issues 

1 . Do hotlinks connect to gopher menus? 

2. Is any of the site restricted by passwords? 

3. Is the page longer than three screens? 

4. Do pages longer than three screens provide anchors? 

5. Does the site consistently differentiate between 
graphics and graphical links? 

6. Does the site have a consistent way of indicating 
text and graphical hotlinks? 

7. Do you need to use the browser to navigate? 

8. Does each page provide an indication of where 
you are in the site? 

9. Does each page indicate what site you are on? 

10. Does the site warn you when you take an outside link? 



Paee I 1 


Paee II Z 


Paee III 1 Paee IV- 


Paee V 


N 6 


Y 


Y N 7 


Y 



N 


Y 


Y 


N 8 


N 


Y 9 


Y 


Y 


Y 


Y 


Y 10 


Y 10 


Y 10 


Y 10 


Y 1( 



Y 


Y 


Y 


Y 


Y 


12/06/96 


10/04/96 


10/09/96 


02/06/97 


01/07/9 



Y 


Y 


Y 


Y 


Y 


Y 


Y 


Y 


Y 


Y 


Y n 


Y n 


Y 


Y 


Y 


Y 


Y 


Y 


Y 


Y 



N 


N 


N 


N 


N 


Y 


Y 


Y 


N 


N 


N 


N 


N 


N 


N 


N/A 


N/A 


N/A 


N/A 


N/A 


N 12 


Y 


Y 


N 


N 


Y 


Y 


Y 


Y 


Y 


N 13 


N 13 


N 13 


N 13 


N 13 


Y 


Y 


Y 


Y 


Y 


Y 


Y 


Y 


Y 


Y 


N 


N 


N 


N 


N 
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Help Issues 



1 . Is there a person or address that can be reached to provide 



feedback or to ask questions? 

2. Are large documents broken up into subsections? 

3. Are indexes provided for large documents and subsections? 

4. Does the site provide a download option for large 








N/A 

N/A 



N/A 

N/A 



N/A 

N/A 



N/A 

N/A 



Y 

N 



documents? 

5. Does the site warn you about file sizes? 

6. Is a text only version of the site available? 








1. Homepage <http://www.stat-usa.gov/stat-usa.html> 

2. STAT-USA Databases <http://www.stat-usa.gov/BEN/databases.htmI> 

3. Business Leads & Procurement Opportunities <http://www.stat-usa.gov/BEN/subject/pocure.htmI> 

4. Selected Publications of Interest <http://www.stat-usa.gov/BEN/publications.html> 

5. ESA Research Series Selected Reports <http://www.stat-usa.gov/BEN/esadocs/esadocs.html> 

6. Index takes you to databases 

7. Two different titled links go to same page 

8. All links with no descriptions; -- 9. The site itself is a service; — 10. In some cases a password is needed 

1 1. The background color changes; - 12. The “Newsletter” switches to frames; - 13. Evaluation was of individual page 
only. 

14. Pages only offer an address which is not hyperlinked, no person’s name; - 15. Some links automatically download 
documents without warning. 

16. It offers a “mostly text version”; - 17. This function may only be appropriate from homepage? 
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3. E. POLICY ASSESSMENT CRITERIA - Summary of All Four Websites 



Policy Criteria 

1 . Does the website provide an e-mail link to a Web 


Census 1 


NIH 2 


nara 1 


STAT-USA 


administrator? 

2. Does the site offer users the opportunity to make 


0 


0 


5 


0 


comments about the site’s content or design? 

3. Is there indication that visitors should expect responses 


2 


5 


4 


5 


to their comments? 

4. Is there a notice of when the website or specific pages 


0 


5 


0 


0 


were last updated? 


5 


0 


5 


5 


5. Does content appear appropriate for public access? 


5 


5 


5 


2 


6. Does content appear appropriate for the agency’s mission? 


5 


5 


5 


5 


7. Does content appear to advocate political positions? 


O 5 


0 


0 


0 


8. Do external links to commercial sites exist? 


2 


0 


4 


1 


9. How appropriate do links to commercial site seem? 


2 


0 


1 


0 


10. Are any data collection mechanisms included on the site? 

11. Do they appear appropriate in terms of the agency’s 


0 


0 


0 


0 


mission? 

12. Does the website provide pointers to alternative means 
of accessing the content (telephone numbers for ordering 


0 


0 


0 


0 


paper copies)? 


0 


0 


0 


0 


13. Are the location aids GILS compliant? 


0 


0 


0 


0 



1. United States Census Bureau <http://vvww.census.gov> 

2. National Institutes of Health <http://www.nih.gov> 

3. National Archives and Records Administration <http://www.nara.gov> 

4. Department of Commerce <http://www.stat-usa.gov> 

5. Page discussed implication of budget limitations on provision of service. Issue discussed in text. 
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3. F. POLICY ASSESSMENT CRITERIA - NIH Website 



Policy Criteria 


Pase I 1 


Page II 2 


Page m l 


Page IV* 


Pase V 


1. 


Does the website provide an e-mail link to a web 
administrator? 


N 


N 


N 


N 


N 


2. 


Does the site offer users the opportunity to make 
comments about the site’s content or design? 


Y 


Y 


Y 


Y 


Y 


3. 


Is there indication that visitors should expect responses 
to their comments? 


Y 


Y 


Y 


Y 


Y 


4. 


Is there a notice of when the website or specific pages 
were last updated? 


N 


N 


N 


N 


N 


5. 


Does content appear appropriate for public access? 


Y 


Y 


Y 


Y 


Y 


6. 


Does content appear appropriate for the agency’s mission? 


Y 


Y 


Y 


Y 


Y 


7. 


Does content appear to advocate political positions? 


N 


N 


N 


N 


N 


8. 


Do external links to commercial sites exist? 


N 


N 


N 


N 


N 


9. 


How appropriate do links to commercial site seem? 


N/A 


N/A 


N/A 


N/A 


N/A 


10. 


Are any data collection mechanisms included on the site? 


N 


N 


N 


N 


N 


11. 


Do they appear appropriate in terms of the agency’s 
mission? 


N/A 


N/A 


N/A 


N/A 


N/A 


12. 


Does the website provide pointers to alternative means 
of accessing the content (telephone numbers for ordering 
paper copies)? 


N 


N 


N 


N 


N 


13. 


Are the location aids GILS compliant? 


N/A 


N/A 


N/A 


N/A 


N/A 



1 . National Institute of Health (NIH) - Homepage <http://vvww.nih.gov> 

2. National Institute of Health — What’s New <http://www.nih.gov/whatsnew.html> 

3. National Institute of Health — Health Information Index <http://www.nih.gov/health/> 

4. National Institute of Health — Scientific Resources <http://www.nih.gov/science/> 

5. National Institute of Health - Institutes, Centers, and Divisions <http://www.nih.gov/icd/> 
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3. G. POLICY ASSESSMENT CRITERIA - NARA Website 



Policy Criteria 


Page t 1 


Paae II 2 


Page III 2 


Page IV- 


Pase V 


1 . 


Does the website provide an e-mail link to a web 
administrator? 


Y 


Y 


Y 


Y 


Y 


2. 


Does the site offer users the opportunity to make 
comments about the site’s content or design? 


Y 6 


N 7 


N 7 


N 7 


N 7 


3. 


Is there indication that visitors should expect responses 
to their comments? 


N 


N 


N 


N 


N 


4. 


Is there a notice of when the website or specific pages 
were last updated? 


Y 


Y 


Y 


Y 


Y 


5. 


Does content appear appropriate for public access? 


Y 


Y 


Y 


Y 


Y 


6 . 


Does content appear appropriate for the agency’s mission? 


Y 


Y 


Y 


Y 


Y 


7. 


Does content appear to advocate political positions? 


N 


N 


N 


N 


N 


8. 


Do external links to commercial sites exist? 


N 


N 


N 


N 


Y 


9. 


How appropriate do links to commercial site seem? 


N/A 


N/A 


N/A 


N/A 


Y 8 


10. 


Are any data collection mechanisms included on the site? 


N/A 


N/A 


N/A 


N/A 


N/A 


11. 


Do they appear appropriate in terms of the agency’s 
mission? 


N/A 


N/A 


N/A 


N/A 


N/A 


12. 


Does the website provide pointers to alternative means 
of accessing the content (telephone numbers for ordering 
paper copies)? 


N 


N 


N 


N 


N 


13. 


Are the location aids GILS compliant? 


N/A 


N/A 


N/A 


N/A 


N/A 



1. Homepage <http://www.nara.gov> 

2. What’s New at NARA <http://www.gov/nara/whatsnew.html> 

3. The Digital Classroom <http://www.nara.gov/nara/digital/cIassroom.html>> 

4. The Japanese Surrender Document <http.V/www.nara.gov/exhall/wwii/surrender.htmI> 

5. NARA and the Public <http://www.nara.gov/nara/whatis/public.Iitml> 

6. Yes, offers a link for Questions, Comments, or Complaints” 

7. Link changed to e-mail only 

8. Yes here indicates appropriate 
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